A cloud rendering method, device, system, electronic device and storage medium

Abstract

The application relates to the computer technical field, and in particular to a cloud rendering method, device and system, electronic equipment and storage medium, the method comprising: receiving an operation instruction triggered by a client for a target application, running the target application according to the operation instruction in a virtual running environment, and obtaining a running result; generating a graphic interface calling instruction corresponding to the running result for a virtual GPU; converting the graphic interface calling instruction into a rendering instruction stream; the instruction format of the rendering instruction stream is a common instruction format among multiple hardware platforms; performing address conversion on the rendering instruction stream to convert virtual address information contained in the rendering instruction stream into a physical offset recognizable by a rendering server; encapsulating the rendering instruction stream after address conversion into a network message according to a preset network protocol, and sending the network message to the rendering server, so that the rendering server returns a rendering result to the client for presentation after executing the rendering instruction stream through a physical GPU.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a cloud rendering method, apparatus, system, electronic device, and storage medium. Background Technology

[0002] Cloud rendering refers to migrating complex graphics computing tasks from the local client to a cloud server cluster for execution, and pushing the rendering results back to the local client in real time in the form of video streams. This enables lightweight terminal devices such as mobile phones, tablets, and smart TVs to run large-scale games smoothly, perform high-precision industrial design, or experience immersive virtual reality applications.

[0003] In related technologies, cloud rendering is mainly achieved through GPU passthrough, a solution executed on a cloud computing virtualization architecture. The essence of this solution is a static binding of one machine (virtual machine) and one card (physical GPU) within the same physical server. This exclusive mode prevents a single physical GPU from simultaneously serving multiple virtual machines, resulting in severe resource fragmentation. For example, even if a virtual machine is only running lightweight rendering tasks, the computing units and video memory of its dedicated physical GPU cannot be reclaimed or reused, leading to extremely low resource utilization.

[0004] On the other hand, since the entire execution chain of the rendering process is encapsulated within the same server, any hardware or software problem that causes the server to fail will cause the entire rendering session hosted on that server to be interrupted, which makes the rendering system less robust.

[0005] Therefore, there is a lack of cloud rendering methods in related technologies that can improve resource utilization and system robustness. Summary of the Invention

[0006] This application provides a cloud rendering method, apparatus, system, electronic device, and storage medium to improve resource utilization and system robustness.

[0007] This application provides a cloud rendering method applied to a computing server in a cloud server cluster. The computing server deploys a network forwarding proxy and a virtual runtime environment containing a virtual GPU. The cloud server cluster also includes a rendering server for executing rendering tasks, and the rendering server deploys a physical GPU. The method includes: Receive operation instructions triggered by the client for the target application, run the target application in the virtual runtime environment according to the operation instructions, and obtain the running results; For the virtual GPU, generate graphics interface call instructions corresponding to the running results; The graphics interface call instructions are converted into a rendering instruction stream; wherein the instruction format of the rendering instruction stream is a common instruction format across multiple hardware platforms. The network forwarding proxy performs address translation on the rendering instruction stream to convert the virtual address information contained in the rendering instruction stream into a physical offset that the rendering server can recognize; the virtual address information is used to indicate the address of each rendering resource associated with the rendering instruction stream within the virtual runtime environment. The rendered instruction stream after address translation is encapsulated into a network packet according to a preset network protocol and sent to the rendering server, so that the rendering server executes the rendered instruction stream through the physical GPU and returns the rendering result to the client for presentation.

[0008] This application provides a cloud rendering method applied to a rendering server in a cloud server cluster. The rendering server is equipped with a physical GPU, and the cloud server cluster also includes a computing server for performing computing tasks. The computing server is equipped with a network forwarding proxy and a virtual runtime environment containing a virtual GPU. The method includes: The system receives network packets sent by the computing server and obtains a rendering instruction stream by parsing the network packets. The network packets are generated as follows: the computing server performs address translation on the rendering instruction stream through a deployed network forwarding proxy, and then encapsulates the address-translated rendering instruction stream according to a preset network protocol to obtain the network packets, thereby converting the virtual address information contained in the rendering instruction stream into physical offsets recognizable by the rendering server. The virtual address information indicates the addresses of each rendering resource associated with the rendering instruction stream within the virtual runtime environment. The rendering instruction stream is generated as follows: the computing server runs the target application in the virtual runtime environment according to the operation instructions triggered by the client for the target application, obtains the running results, and generates graphics interface call instructions corresponding to the running results for the virtual GPU, and converts the graphics interface call instructions to obtain the rendering instruction stream. The instruction format of the rendering instruction stream is a common instruction format across multiple hardware platforms. The rendering instruction stream is decoded to obtain executable drawing instructions; The executable drawing instructions are executed by the physical GPU to obtain the rendering result; The rendering result is returned to the client so that the client can render the rendering result.

[0009] This application provides a cloud rendering device applied to a computing server in a cloud server cluster. The computing server deploys a network forwarding proxy and a virtual runtime environment containing a virtual GPU. The cloud server cluster also includes a rendering server for executing rendering tasks, and the rendering server deploys a physical GPU. The device includes: A computing unit is configured to receive operation instructions triggered by a client for a target application, run the target application according to the operation instructions in the virtual runtime environment, and obtain the running results; generate graphics interface call instructions corresponding to the running results for the virtual GPU; and convert the graphics interface call instructions into a rendering instruction stream; wherein the instruction format of the rendering instruction stream is a common instruction format across multiple hardware platforms. The communication unit is configured to perform address translation on the rendering instruction stream via the network forwarding proxy, so as to convert the virtual address information contained in the rendering instruction stream into a physical offset that can be recognized by the rendering server; the virtual address information is used to indicate the address of each rendering resource associated with the rendering instruction stream within the virtual runtime environment; the address-translated rendering instruction stream is encapsulated into a network packet according to a preset network protocol, and the network packet is sent to the rendering server, so that the rendering server executes the rendering instruction stream through the physical GPU and returns the rendering result to the client for presentation.

[0010] Optionally, the computing unit is specifically used to: call an address mapping function to convert the virtual address information contained in the rendering instruction stream into an address pointer accessible by the network forwarding agent; The rendering instruction stream is read through the address pointer, and the virtual resource identifier and virtual offset of each rendering resource associated with the rendering instruction stream are extracted; the virtual resource identifier is the resource identifier in the virtual runtime environment, and the virtual offset represents the offset of the corresponding rendering resource based on the resource start address in the virtual runtime environment; Based on the virtual resource identifier and virtual offset of each rendering resource, the global resource identifier and physical offset of each rendering resource are determined; the physical offset represents the offset of the corresponding rendering resource from the physical storage start address in the rendering server; the physical storage start address is located based on the global resource identifier of the corresponding rendering resource. Based on the global resource identifier and physical offset of each rendering resource, the rendering instruction stream after address translation is obtained.

[0011] Optionally, the network forwarding proxy is pre-bound to the virtual instruction queue and virtual resource queue created by the virtual GPU; before sending the network packet to the rendering server, the communication unit is further configured to: Write the rendering instruction stream into the virtual instruction queue, write each rendering resource associated with the rendering instruction stream into the virtual resource queue, and trigger a notification message for the network forwarding proxy; When sending the network message to the rendering server, the communication unit is specifically used for: After receiving the notification message, the network forwarding agent reads the rendering instruction stream and each rendering resource, and encapsulates the rendering instruction stream and each rendering resource into a network packet according to a preset network protocol; The network message is sent to the rendering server.

[0012] The network message includes command messages and resource messages; the computing unit is specifically used for: The rendering instruction stream is encapsulated into instruction packets according to a preset network protocol, and each rendering resource is encapsulated into a resource packet according to a preset network protocol. The communication unit is specifically used for: The instruction message is sent to the rendering server via the instruction channel; The resource message is sent to the rendering server through the resource channel; The instruction channel and the resource channel are pre-established communication channels between the network forwarding agent and the rendering server.

[0013] Optionally, the cloud server cluster includes multiple rendering servers, and the communication unit is specifically used to: select a rendering server that meets the preset resource requirements from the multiple rendering servers based on the latest resource usage status of each of the multiple rendering servers. The network message is sent to the selected rendering server.

[0014] Optionally, before receiving the operation command triggered by the client for the target application, the computing unit is further configured to: Based on the graphics attributes of each application in the virtual runtime environment, including the target application, determine the set of virtual GPU functions that need to be deployed; Based on the aforementioned feature set, a virtual GPU is registered in the kernel of the computing server, and the virtual GPU is initialized. The initialized virtual GPU is mounted into the virtual runtime environment.

[0015] This application provides a cloud rendering device applied to a rendering server in a cloud server cluster. The rendering server is equipped with a physical GPU. The cloud server cluster also includes a computing server for performing computing tasks. The computing server is equipped with a network forwarding proxy and a virtual runtime environment containing a virtual GPU. The device includes: A communication unit is configured to receive network packets sent by the computing server and obtain a rendering instruction stream by parsing the network packets. The network packets are generated as follows: the computing server performs address translation on the rendering instruction stream through a deployed network forwarding proxy, and then encapsulates the address-translated rendering instruction stream according to a preset network protocol to obtain the network packets, thereby converting the virtual address information contained in the rendering instruction stream into physical offsets recognizable by the rendering server. The virtual address information indicates the addresses of each rendering resource associated with the rendering instruction stream within the virtual runtime environment. The rendering instruction stream is generated as follows: the computing server runs the target application in the virtual runtime environment according to the operation instructions triggered by the client for the target application, obtains the running results, and generates graphics interface call instructions corresponding to the running results for the virtual GPU, and converts the graphics interface call instructions into the rendering instruction stream. The instruction format of the rendering instruction stream is a common instruction format across multiple hardware platforms. The rendering unit is used to decode the rendering instruction stream to obtain executable drawing instructions; and to execute the executable drawing instructions through the physical GPU to obtain the rendering result. The communication unit is also used to return the rendering result to the client so that the client can present the rendering result.

[0016] Optionally, the communication unit is specifically used for: The network forwarding agent receives instruction packets from the network packets via an instruction channel and resource packets from the network packets via a resource channel; wherein the instruction channel and the resource channel are pre-established communication channels between the network forwarding agent and the rendering server. The rendering unit is specifically used for: The rendering instruction stream is obtained by parsing the instruction message, and the rendering resources associated with the rendering instruction stream are obtained by parsing the resource message.

[0017] Optionally, the rendering unit is specifically used for: Based on the rendering resources, the executable drawing instructions are executed through the physical GPU to obtain the rendering result.

[0018] This application provides a cloud rendering system, including a client, a computing server, and a rendering server. The computing server is equipped with a network forwarding proxy and a virtual runtime environment containing a virtual GPU, and the rendering server is equipped with a physical GPU. The client is used to send operation instructions triggered by the target application to the computing server; The computing server is configured to receive the operation instructions, run the target application according to the operation instructions in the virtual runtime environment, obtain the running results, generate graphics interface call instructions corresponding to the running results for the virtual GPU, and convert the graphics interface call instructions into a rendering instruction stream; the instruction format of the rendering instruction stream is a common instruction format across multiple hardware platforms; the rendering instruction stream is address-translated through the network forwarding proxy to convert the virtual address information contained in the rendering instruction stream into physical offsets recognizable by the rendering server; the virtual address information is used to indicate the addresses of each rendering resource associated with the rendering instruction stream within the virtual runtime environment; the address-translated rendering instruction stream is encapsulated into a network packet according to a preset network protocol and sent to the rendering server; The rendering server is configured to receive network packets sent by the computing server and obtain a rendering instruction stream by parsing the network packets; decode the rendering instruction stream to obtain executable drawing instructions; execute the executable drawing instructions through the physical GPU to obtain a rendering result; and return the rendering result to the client. The client is also used to present the rendering result.

[0019] Optionally, there are multiple rendering servers, and the system further includes a resource scheduler, which is used for: Based on the latest resource usage status of each of the multiple rendering servers, a rendering server that meets the preset resource requirements is selected from the multiple rendering servers, so that the computing server sends the network packet to the selected rendering server.

[0020] An electronic device provided in this application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any of the cloud rendering methods described above.

[0021] This application provides a computer-readable storage medium including a computer program. When the computer program is run on an electronic device, the computer program is used to cause the electronic device to perform the steps of any of the cloud rendering methods described above.

[0022] This application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. When the processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the above-described cloud rendering methods.

[0023] In this embodiment, the computing server does not have a physical GPU; instead, it uses a virtual GPU in a virtual runtime environment and sends rendering command streams to the physical GPU on a remote rendering server for execution. This technical feature enables physical decoupling of computing and rendering, allowing computing and rendering resources to be planned and scaled elastically independently. This avoids the resource contention or idleness issues found in traditional all-in-one servers, significantly improving overall resource utilization. Furthermore, faults between the computing server and the rendering server are isolated, meaning a single resource layer failure does not affect the operation of other service layers, thereby improving the overall robustness and availability of the system.

[0024] Furthermore, by converting graphics interface call instructions into a rendering instruction stream that is common across multiple hardware platforms, and by enabling the rendering server to understand and execute the rendering operations indicated by the rendering instruction stream through address translation, transparent compatibility with different underlying GPU hardware can be achieved. Rendering can be completed on any physical GPU model without modifying the graphics framework or driver of the target application, ensuring broad ecosystem compatibility.

[0025] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0026] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of an application scenario in an embodiment of this application; Figure 2 A flowchart illustrating the implementation of a cloud rendering method provided in this application embodiment; Figure 3 This is a schematic diagram of the network message sending process provided in an embodiment of this application; Figure 4 This is another schematic diagram of a network message sending process provided in an embodiment of this application; Figure 5A flowchart illustrating the implementation of a cloud rendering method provided in this application embodiment; Figure 6 This is a schematic diagram of a cloud rendering system provided in an embodiment of this application; Figure 7 This is a schematic diagram of dynamic resource scheduling provided in an embodiment of this application; Figure 8 This is a schematic diagram of the overall architecture of the cloud rendering method provided in the embodiments of this application; Figure 9 This is a schematic diagram of the overall process of a cloud rendering method provided in an embodiment of this application; Figure 10 This is a schematic diagram of cloud rendering methods in related technologies; Figure 11 A schematic diagram illustrating the rendering process of a cloud rendering method in a given scenario, provided in the application embodiment; Figure 12 This is a diagram comparing the idle computing power fragments of cloud-based games and cloud-based mobile games in related technologies; Figure 13 A schematic diagram illustrating the rendering process of a cloud rendering method in another scenario provided in the application embodiment; Figure 14 This is a schematic diagram of the composition structure of a cloud rendering device according to an embodiment of this application; Figure 15 This is a schematic diagram of the composition structure of a cloud rendering device according to an embodiment of this application; Figure 16 This is a schematic diagram illustrating the rendering effect provided in an embodiment of this application; Figure 17 This is a schematic diagram of the hardware structure of an electronic device using an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0028] The following describes some of the concepts involved in the embodiments of this application.

[0029] Cloud rendering: a graphics processing service based on the cloud computing paradigm. Its core feature is offloading graphics rendering tasks, which are extremely demanding in terms of computation and input / output (I / O) resources, from the user's terminal device to a remote cloud server cluster for execution. In this scenario, the terminal device is only responsible for receiving user input commands and decoding and displaying the encoded video stream returned by the cloud server cluster. Cloud gaming is an application of cloud rendering, which places the complete game logic calculation and graphics rendering on a cloud server cluster. Terminal devices can obtain game visuals through network streaming, thereby achieving a cross-platform, low-barrier, high-quality gaming experience.

[0030] Open Graphics Library (OpenGL): A cross-language, cross-platform application programming interface specification for 2D and 3D vector graphics. It provides a standardized interface for upper-layer applications to access the rendering capabilities of graphics hardware and is the industry cornerstone for hardware-accelerated rendering. This interface is typically used to interact with image processing units to achieve hardware acceleration. For embedded systems (such as mobile phones and tablets), a simplified version of OpenGL exists, called OpenGL for Embedded Systems (OpenGL ES). Optionally, the graphics interface call instructions in this application embodiment can refer to OpenGL calls or OpenGL ES calls.

[0031] Kernel-based Virtual Machine (KVM): A full virtualization solution integrated into the Linux kernel. KVM directly utilizes the processor's hardware virtualization capabilities to efficiently and securely virtualize and manage core computing resources such as CPU and memory.

[0032] Quick Emulator (QEMU): An open-source, powerful system emulator and virtualizer. In a typical virtualization environment that works with KVM, KVM is responsible for the underlying CPU and memory virtualization acceleration, while QEMU is responsible for emulating a complete computer system, including disks, networks, graphics devices, and various input / output peripherals, providing a runtime environment for the virtual machine.

[0033] Virtual GPU: A device that presents graphics processing capabilities to a virtual runtime environment through software or paravirtualization. Optionally, its full name can be Virtual Graphics Processing Device based on the VirtIO protocol (VirtIO-GPU). In the Virgl virtualized graphics technology stack, VirtIO-GPU is provided by QEMU and is responsible for receiving graphics instructions generated by the graphics driver inside the virtual machine, encapsulating them, and forwarding them to the Virgl Renderer library on the host machine for subsequent processing.

[0034] Virgl Renderer: One of the core components of the Virgl virtualized graphics technology stack, it typically runs in the host user space as part of the QEMU process. Its core function is to act as the backend driver for VirtIO-GPU, receiving and parsing the Virgl protocol instruction stream passed through from the virtual machine, and translating or converting it into standard host-side OpenGL calls, thereby driving the actual graphics hardware (or software implementation) to complete the rendering.

[0035] Mesa graphics library: An open-source 3D graphics library that implements various graphics application programming interfaces (APIs), including OpenGL. Within the VirGL technology stack, it exists in two key instances: 1) In-Virtual Machine Mesa Graphics Library: Integrates the VirGL driver, responsible for intercepting OpenGL calls from in-virtual machine applications and translating them into VirGL protocol instructions. 2) Host Machine Mesa Graphics Library: Acts as the user-mode driver for the physical GPU hardware, responsible for executing the final, local or remote graphics rendering commands.

[0036] Computing server: In this embodiment of the application, a computing server refers to a server used to perform computing tasks. A computing server may only be equipped with a CPU, that is, it may not contain a physical GPU. For example, for a computing server for cloud application services under the Android platform, it may contain an Android virtual runtime environment (such as an Android virtual container) for processing application logic, user commands and audio, and generating rendering instruction streams.

[0037] Rendering server: In this embodiment of the application, a rendering server refers to a server used to perform rendering tasks. The rendering server is equipped with a CPU and a physical GPU, receives rendering instruction streams from the computing server, accelerates graphics rendering and video encoding through the physical GPU, and outputs a video stream.

[0038] Virtual runtime environment: An isolated runtime environment built using technologies such as containerization (e.g., Docker), virtual machines, or sandboxes. In this embodiment, applications requiring cloud rendering services, such as cloud mobile games and cloud gaming, can run within the virtual runtime environment. Specifically, for the Android platform, the instance form of the virtual runtime environment includes, but is not limited to, lightweight virtualization containers based on the Android virtualization framework; for the Windows platform, the instance form of the virtual runtime environment includes, but is not limited to, sandboxes.

[0039] Operation commands: After a user performs touch, click, or other operations on a target application on a terminal device (such as a mobile phone, tablet, or computer), these operations can be encoded into operation commands and uploaded to the computing server via a low-latency data channel. This allows the computing server to perform the same operations on the target application as the user within the virtualized runtime environment. For example, a low-latency data channel could be a data channel for Web Real-Time Communication (WebRTC).

[0040] Rendering command stream: A hardware-independent intermediate representation of rendering commands, including but not limited to Virgl protocol command streams. In the specific context of this application, it refers to a sequence of graphics rendering commands conforming to the Virgl protocol specification, generated by the graphics driver inside a virtual machine or virtual container and transmitted through the VirtIO-GPU channel. This command stream is the core data carrier for achieving decoupling between rendering and computation in this application. It contains the geometry, texture, state, and shader information necessary to reconstruct a frame. Compared to the encoded video stream, it has the characteristics of being portable, capable of being executed remotely, and having relatively flexible bandwidth requirements.

[0041] Physical GPU: In contrast to virtual GPU, physical GPU is the actual hardware graphics processing unit chip installed on the rendering server.

[0042] Executable drawing instructions: Instructions executable by the physical GPU of the rendering server for performing graphics rendering. Optionally, in this embodiment, the executable drawing instructions can be graphics interface call instructions. For example, the executable drawing instructions can also be OpenGL calls or OpenGL ES calls.

[0043] The terms "first" and "second" used in this document are for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0044] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0045] like Figure 1 The diagram shown is an application scenario illustration of an embodiment of this application. The application scenario diagram includes a cloud server cluster 100 and a terminal device 110, wherein the cloud server cluster 100 may include a computing server 120 and a rendering server 130.

[0046] In this embodiment, the terminal device 110 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances (such as smart speakers, smart TVs, etc.), smart wearable devices (such as smartwatches), and in-vehicle terminals. The terminal device may have a client installed on it that is related to the target application that needs to provide cloud rendering services. The client may be software (such as a browser, the target application's software, etc.), or a webpage, a mini-program, etc. The computing server 120 and the rendering server 130 are the backend servers corresponding to the software, webpage, mini-program, etc. The computing server 120 may be a server used to execute the computing tasks corresponding to the target application, and the rendering server 130 may be a server used to execute the rendering tasks corresponding to the target application. This application does not make any specific limitations.

[0047] The computing server 120 and the rendering server 130 can be independent physical servers, or server clusters or distributed systems composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0048] It should be noted that the cloud rendering method in each embodiment of this application can be executed by an electronic device, which can be a terminal device 110, a computing server 120 or a rendering server 130. The method can be executed jointly by the terminal device 110, the computing server 120 or the rendering server 130.

[0049] In one alternative implementation, computing server 120 is equipped with a network forwarding agent and a virtual runtime environment containing a virtual GPU, while rendering server 130 is equipped with a physical GPU.

[0050] Terminal device 110, acting as a client, can respond to user-triggered operation commands for a target application and send the command to computing server 120.

[0051] The computing server 120 can receive operation instructions and, upon receiving them, executes the target application within the virtual runtime environment to obtain the execution results. Then, for the virtual GPU, it generates graphics interface call instructions corresponding to the execution results and converts these instructions into a rendering instruction stream. Next, a network forwarding proxy performs address translation on the rendering instruction stream, converting the virtual address information contained within the stream into physical offsets recognizable by the rendering server. The address-translated rendering instruction stream is then encapsulated into network packets according to a preset network protocol and sent to the rendering server. The instruction format of the rendering instruction stream is a common format across multiple hardware platforms; the virtual address information indicates the addresses of each rendering resource associated with the rendering instruction stream within the virtual runtime environment. In one alternative implementation, the rendering server 130 can receive network packets sent by the computing server 120 and obtain a rendering instruction stream by parsing the network packets. Then, by decoding the rendering instruction stream, executable drawing instructions are obtained, and the executable drawing instructions are executed by the deployed physical GPU to obtain the rendering result, which is then returned to the terminal device 110.

[0052] After receiving the rendering result, the terminal device 110 can display the rendering result on its display interface.

[0053] In one alternative implementation, the terminal device 110 and the server 120 can communicate via a communication network.

[0054] In one alternative implementation, the communication network is a wired network or a wireless network.

[0055] It should be noted that, Figure 1 The examples shown are merely illustrative; in reality, the number of terminal devices, computing servers, and rendering servers is unlimited and is not specifically limited in this embodiment.

[0056] Furthermore, the embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving. It is primarily used for remote hardware rendering acceleration in various scenarios. Several examples are briefly listed below: Application Scenario 1: Remote hardware rendering acceleration in cloud gaming.

[0057] When players enjoy large-scale open-world games on thin and light laptops, tablets, or mobile phones without dedicated graphics cards, the cloud rendering method provided in this application can be used to achieve remote hardware rendering acceleration. Specifically, the terminal device receives operation commands input by the player via keyboard, mouse, or touch, and sends these commands over the network to a computing server equipped with a Central Processing Unit (CPU) but without a GPU. The computing server runs game logic, audio processing, and non-graphics-related physics calculations, while simultaneously generating a rendering command stream. This rendering command stream can be transmitted over the network to a rendering server equipped with a physical GPU. The rendering server parses the rendering command stream and invokes the physical GPU to complete the entire graphics rendering pipeline, ultimately pushing the encoded video stream to the user terminal for decoding and display.

[0058] For example, in a cloud-based open-world role-playing game, the player's movement, skill activation, and interaction with the environment are all independently calculated by the CPU threads of server A, generating rendering instruction streams. Server B then uses its deployed physical GPUs to render the scene from the player's perspective based on these rendering instruction streams. In this scenario, the GPU resource pool of server B can be scaled up or down on demand, decoupled from the number of CPU instances on server A, thereby achieving independent elastic scaling of computation and rendering, and flexibly expanding the cloud server cluster.

[0059] Application Scenario 2: Hybrid deployment of cloud-based games and cloud-based mobile games resources.

[0060] Currently, cloud gaming services exist in two main forms: cloud-based mobile gaming and cloud-based mobile gaming. Cloud-based mobile gaming primarily uses the x86 architecture, providing a Windows gaming environment through virtual machines or sandbox technology. Cloud-based mobile gaming primarily uses the ARM architecture, relying on containerization technology to run Android game applications. Due to the fundamental differences in their underlying instruction sets (ARM and x86), they are typically deployed in isolation in traditional solutions, forming independent GPU resource pools. This results in GPU resources becoming silos between different services, leading to severe resource fragmentation and idle waste. The cloud rendering method provided in this application can migrate rendering instruction streams over the network to a GPU cluster that meets resource requirements for processing. This allows cloud-based mobile gaming and cloud-based mobile gaming to coexist on the same platform and dynamically allocate resources based on actual load conditions, thereby achieving more efficient resource utilization.

[0061] For example, a cloud gaming service provider operates both a mobile cloud version and a cloud-based mobile version of game A. During peak daytime hours, the number of users in the mobile cloud version surges, while the number of users in the cloud-based mobile version is relatively low. In this situation, by using the cloud rendering method provided in this application, the service provider's resource scheduling platform can distribute the rendering instruction stream generated in the mobile cloud version container to the rendering server in the cloud-based mobile version for execution. At this time, the rendering servers that were originally allocated to the cloud-based mobile version but were idle are scheduled to process the rendering instruction stream of the mobile cloud version, thereby filling the computing power gap of the mobile cloud version. This achieves logical pooling and dynamic sharing of GPU resources between the mobile cloud version and the cloud-based mobile version, effectively solving the problem of computing power fragmentation caused by differences in business peak and valley periods and architectural isolation.

[0062] Application Scenario 3: Accelerating remote hardware rendering in cloud-based computer-aided design (CAD) collaboration.

[0063] When design teams collaboratively edit complex 3D mechanical models using low-computing-power terminal devices, the cloud rendering method provided in this application can be used to accelerate remote hardware rendering. For example, a cloud server cluster may include server A and server B. Server A is a GPU-less computing server used to run the geometric kernel of CAD software, responsible for computationally intensive tasks such as Boolean operations, constraint solving, parametric modification, and topology reconstruction, and converting model change results into rendering command streams. Server B is a rendering server containing a physical GPU, which can call the physical GPU to perform high-quality rendering such as anti-aliasing and ambient occlusion after receiving the rendering command stream, and encode the rendering results into a video stream to be sent back to each low-computing-power terminal device.

[0064] For example, an automotive chassis engineer modifies the length parameters of a suspension link on a terminal device. Server A's CPU immediately recalculates the interference relationships and assembly constraints of all related parts, generating an incremental rendering command stream that only describes the changed parts (such as the new position of the link or the highlighted areas of the contact surface). Upon receiving the incremental rendering command stream, Server B can utilize its GPU to re-render only the dirty areas and merge them into the cached frame, avoiding a full frame redraw. Throughout this process, the engineer's terminal device is only used to display the video stream and receive mouse input, without participating in any graphics calculations. Server A and Server B communicate via a dedicated message queue, and the rendering command stream employs a differential update strategy, keeping network bandwidth usage consistently low.

[0065] It should be noted that the application scenarios listed above are just simple examples. In addition, other scenarios such as sensor data visualization in autonomous driving simulation testing, full-body motion capture rendering in metaverse virtual conferences, and real-time analysis of remote sensing images in smart agriculture are also applicable to the embodiments of this application, and will not be described in detail here.

[0066] Furthermore, it should be emphasized that in the specific implementation of this application, if the relevant data of the object is involved, when the above embodiments of this application are applied to specific products or technologies, permission or consent from the object is required, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0067] The cloud rendering method provided by the exemplary embodiments of this application will be described below with reference to the accompanying drawings and the application scenarios described above. It should be noted that the application scenarios described above are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0068] See Figure 2 The diagram shown is a flowchart of a cloud rendering method provided in this application embodiment. The execution entity in this method is a computing server, and the specific implementation process of this method is as follows: S21~S25: S21: Receive the operation command triggered by the client for the target application, run the target application in the virtual runtime environment according to the operation command, and obtain the running result.

[0069] The computing server can receive operation instructions and run the target application in the virtual runtime environment deployed on the computing server. It processes the operation instructions, completes the logical calculations of the target application, and obtains the running result of the target application. This result can represent the logical state obtained after the logical calculations. The operation instructions are encoded by the client after the user performs at least one operation event, such as clicking or dragging. Optionally, after receiving the operation instructions, the client uploads them to the computing server via a low-latency channel (such as WebRTC's DataChannel).

[0070] Optionally, the target application and operation instructions can differ in different application scenarios. For example, in a cloud gaming scenario, the target application can be a game application, and the operation instructions can be instructions for controlling a game character to perform a specific action, such as controlling a game character to release skill A by clicking on skill A; in a cloud-based computer-aided design collaboration scenario, the target application can be a computer-aided design application, and the operation instructions can be instructions for editing complex 3D mechanical models. This application does not impose specific limitations in this regard.

[0071] In some embodiments, before the computing server receives the operation instructions triggered by the client for the target application, it can first prepare the virtual runtime environment and register the virtual GPU. Specifically, during the preparation of the virtual runtime environment, a virtualized runtime environment instance of the appropriate type can be deployed on the computing server according to the type of the target operating system. For example, when the target operating system is Android, an Android virtual container can be started on the host based on container technology (such as Docker), or an Android virtual machine can be started based on virtual machine technology, and the started Android virtual container or Android virtual machine can be used as the Android virtual runtime environment. When the target operating system is Windows, a Windows virtual machine can be started on the computing server based on virtual machine technology, or a sandbox instance can be started based on sandbox technology, and the started Windows virtual container or sandbox instance can be used as the Windows virtual runtime environment. The target operating system can be determined based on the operating system version of the target application. For example, if the target application is an Android mobile game version of game A, then the target operating system can be determined to be Android. This application does not impose specific limitations on this.

[0072] After completing the environment preparation, virtual GPUs can be registered. During implementation, the feature set of the virtual GPUs to be deployed can be determined based on the graphics attributes of each application in the virtual runtime environment, including the target application. Based on the feature set, the virtual GPUs are registered in the kernel of the computing server, initialized, and then mounted into the virtual runtime environment.

[0073] Specifically, taking Android virtual containers as an example of a virtual runtime environment, the feature set of the virtual GPU can be determined based on the graphics attributes required by each application within the container. These graphics attributes may include, but are not limited to, OpenGL ES version, texture format, and other attributes.

[0074] For example, consider two applications running simultaneously within an Android virtual container: Application A declares that it requires OpenGL ES 3.0 support, and its texture format must include ETC2 compressed texture format and RGBA16F (16-bit half-precision floating-point per channel RGBA format); Application B declares that it requires OpenGL ES 2.0 support, and its texture format only needs to be RGBA8 (8-bit per channel RGBA format) and RGB565 format (16-bit RGB565). Based on these graphics attributes, we can determine that the virtual GPU's feature set should at least include the OpenGL ES 3.1 API profile (to cover Application A's ES 3.0 requirements and provide expansion space), a texture format support set (this support set should at least include ETC2, RGBA16F, RGBA8, and RGB565), and resource limits such as a depth buffer of at least 24 bits and a maximum texture size of at least 4096.

[0075] Then, after determining the feature set, a VirtIO-GPU device can be registered and initialized in the kernel of the computing server (host machine) based on this feature set. This device is the virtual GPU mentioned above. After initialization, the virtual GPU can be mounted to the / dev / dri / directory of the Android virtual container, so that the graphics stack within the Android virtual container can correctly recognize and use the virtual GPU.

[0076] The graphics stack within the Android virtual container may include, but is not limited to, the Mesa graphics library with a Virgl driver and the libdrm interface. The libdrm interface provides a user-space interface for interacting with the kernel-level Direct Rendering Manager (DRM) subsystem, used to manage low-level operations such as memory allocation, frame buffer submission, and mode settings. In this embodiment, the Mesa graphics library can interact with the kernel graphics driver through the libdrm interface. This kernel graphics driver refers to the driver of the virtual GPU, i.e., the driver of the aforementioned VirtIO-GPU device.

[0077] S22: For the virtual GPU, generate graphics interface call instructions corresponding to the running results.

[0078] After obtaining the running results, the virtual runtime environment in the computing server can generate graphics interface call instructions corresponding to the running results through the graphics engine in the virtual runtime environment for the virtual GPU. These graphics interface call instructions include, but are not limited to, OpenGL ES calls and OpenGL calls.

[0079] This graphics interface call instruction is used to instruct the virtual GPU to perform specific rendering operations to generate rendering results corresponding to the running results. Rendering operations may include, but are not limited to, drawing geometry, setting rendering state, managing texture and shader resources, and initiating drawing calls.

[0080] However, since virtual GPUs do not possess actual graphics rendering capabilities, it is necessary to serialize and encapsulate the graphics interface call instructions into a custom, platform-independent rendering instruction stream, which is then sent to a rendering server equipped with a physical GPU to perform the rendering operation. The specific conversion process will be detailed in S23.

[0081] S23: Convert the graphics interface call instruction into a rendering instruction stream; wherein the instruction format of the rendering instruction stream is a common instruction format across multiple hardware platforms.

[0082] The computing server can intercept the generated graphics interface call instructions and convert and encapsulate them into a platform-independent rendering instruction stream. That is, the instruction format of the rendering instruction stream can be used across multiple hardware platforms. The rendering instruction stream can include the geometry, texture, state, and shader information necessary to reconstruct a frame of the target application.

[0083] Taking the Android virtual container as an example of a virtual runtime environment, in specific implementation, the Mesa graphics library containing the Virgl driver contained in the Android virtual container can intercept the OpenGL ES calls of the target application and convert them into a platform-independent Virgl protocol instruction stream. The Virgl protocol instruction stream here is the rendering instruction stream mentioned above.

[0084] Besides the virgl protocol instruction stream, the rendering instruction stream can also be other intermediate instruction streams generated by serializing graphics interface call instructions (such as OpenGL ES calls). For example, it could be a custom rendering instruction sequence formed by encoding OpenGL ES call parameters and associated data, or a generic graphics state instruction stream decoupled from specific physical GPU vendor drivers, obtained through a graphics intermediate layer. This rendering instruction stream abstracts the common semantics of vertex processing, rasterization, and fragment shading stages, and can be replayed and executed on different types of physical GPUs. This application does not impose specific restrictions on the specific encoding format of the rendering instruction stream.

[0085] In this embodiment, the decoupling of computation and rendering is achieved by transmitting rendering command streams. Compared to the method of transmitting encoded video streams in cloud rendering methods of related technologies, this method of transmitting rendering command streams has the advantages of portability, remote execution, and relatively flexible bandwidth requirements. Moreover, since this implementation does not modify the Android graphics framework (such as SurfaceFlinger), but instead abstracts and shields chip architecture differences (ARM / x86) through rendering command streams, it achieves seamless migration in heterogeneous hardware environments, effectively improving cross-platform compatibility.

[0086] Furthermore, the instruction format of the rendering instruction stream is universal across multiple hardware platforms. Therefore, when distributing rendering tasks, differences in instruction sets can be ignored, and the rendering instruction stream can be directly migrated over the network to a suitable rendering server for processing. Taking a hybrid deployment scenario of cloud-based mobile games and cloud gaming resources as an example, since the instruction format of the rendering instruction stream is compatible with both x86 and ARM instruction sets, rendering tasks for cloud gaming can be distributed to servers running on the cloud gaming platform. This allows cloud gaming and cloud-based mobile games to coexist on the same platform and dynamically allocate resources based on actual load conditions, resulting in more efficient resource utilization.

[0087] S24: Through a network forwarding proxy, address translation is performed on the rendering command stream to convert the virtual address information contained in the rendering command stream into physical offsets that the rendering server can recognize; the virtual address information is used to indicate the addresses of each rendering resource associated with the rendering command stream within the virtual runtime environment.

[0088] The computing server can first obtain the rendering instruction stream in the virtual runtime environment, and then transmit the rendering instruction stream to the rendering server so that the rendering server can use its physical GPU to perform hardware-accelerated rendering and generate rendering results.

[0089] In practice, after obtaining the rendering command stream in the virtual runtime environment, the Mesa graphics library can also interact with the kernel's virtual GPU driver through the libdrm interface. That is, the Mesa graphics library can send the rendering command stream and the rendering resources (such as texture data) involved in the rendering command stream to the kernel-mode VirtIO-GPU driver. After receiving the rendering command stream and rendering resources, the kernel-mode VirtIO-GPU driver can send the rendering command stream and rendering resources to the rendering server via network transmission.

[0090] In order for the rendering server receiving the rendering command stream to understand and execute the rendering operations indicated by the rendering command stream, address translation is required to convert the address information from the perspective of the virtual runtime environment into a format that the rendering server can understand. That is, the virtual offset representing the relative address difference within the address space needs to be converted into the physical offset representing the byte offset of the resource data.

[0091] For example, when the virtual runtime environment is an Android virtual container, the virtual address information inside the container can be converted into a format that the receiving end can recognize: resource identifier + physical offset + actual data. In this case, after receiving the rendering instruction stream, the rendering server can parse it and find the location of the rendering resource according to the physical offset, thereby performing the corresponding rendering operation.

[0092] The address translation operation described above converts virtual address information within the virtual runtime environment into physical offsets recognizable by the rendering server. This decouples computation from rendering, laying the foundation for subsequent resource pooling and cost optimization.

[0093] In some embodiments, the network forwarding proxy can be a proxy, inserted at a key node in the data path of the Virgl graphics technology stack: the kernel-mode VirtIO-GPU driver. This allows the network forwarding proxy to intercept platform-independent Virgl rendering instruction streams that should be processed by the local Virgl Renderer library and transmit them to a remote rendering server via a low-latency network. This achieves a novel architecture that decouples the computation-rendering-encoding three-stage pipeline from physical components, completely breaking the tight coupling between computation and rendering deployed on the same machine.

[0094] In the above implementation, the decoupling of computation and rendering ensures that failures of the compute servers (as compute nodes) and rendering servers (as rendering nodes) within the cloud server cluster do not affect each other, thereby enhancing the availability and robustness of the cloud rendering system. Furthermore, after decoupling, multiple rendering servers can be deployed to form an independent rendering cluster for all compute nodes without GPUs to share on demand, significantly improving resource utilization, reducing hardware costs, and achieving GPU resource pooling and elastic scaling. The sharing methods of multiple rendering servers will be described in detail later and will not be repeated here.

[0095] In one optional implementation, when the computing server performs address translation on the rendering command stream via the network forwarding proxy, it can do the following: call an address mapping function to convert the virtual address information contained in the rendering command stream into address pointers accessible to the network forwarding proxy. The rendering command stream can be read using these address pointers, and the virtual resource identifier and virtual offset of each rendering resource associated with the rendering command stream can be extracted. The virtual resource identifier is the resource identifier within the virtual runtime environment, and the virtual offset represents the offset of the corresponding rendering resource from its resource start address within the virtual runtime environment. The network forwarding proxy then determines the global resource identifier and physical offset of each rendering resource based on its respective virtual resource identifier and virtual offset. The physical offset represents the offset of the corresponding rendering resource from its physical storage start address in the rendering server, which is located based on the global resource identifier of the corresponding rendering resource. Based on the global resource identifier and physical offset of each rendering resource, the address-translated rendering command stream is obtained.

[0096] For example, the network forwarding proxy can call the map_guest function to use the memory map (mmap) mechanism driven by the virtual input / output loopback to convert the container guest physical address (GPA) into an address pointer that can be directly read and written in the proxy's own virtual address space. Here, GPA is the virtual address information mentioned above, and the address pointer is the address assigned to the proxy process by the computing server (host machine) through the mapping operation.

[0097] Specifically, the virtio-loopback driver establishes a pair of shared memory regions between the host and the guest machine, and exchanges memory offset and length information through a virtual (vring) queue. The proxy process executes the mmap system call on this shared region to map the host machine physical page frame corresponding to the guest machine's physical address to the address space of the proxy process, thereby enabling direct, zero-copy access to the guest machine's memory. The host machine mentioned above can be understood as the computing server in this embodiment, and the guest machine can be understood as the virtual runtime environment on the computing server, i.e., a virtual machine or container.

[0098] After obtaining the address pointer, the proxy can parse the rendering instruction stream and extract the virtual resource identifiers and virtual offsets of each rendering resource referenced in the stream. These virtual resource identifiers can be understood as indexes of rendering resources defined within the virtual runtime environment. Then, based on the resource mapping table pre-established during the rendering resource creation phase, the virtual resource identifiers and virtual offsets can be converted into global resource identifiers and physical offsets, thus obtaining the address-translated rendering instruction stream.

[0099] By implementing the above methods, converting virtual offsets in the virtual runtime environment into physical offsets in the network protocol allows the rendering server, acting as the receiver of the rendering command stream, to determine the read location of each rendering resource based on its physical offset and the physical storage start address located by the global resource identifier. This enables the rendering server to understand the address-converted rendering command stream. With the rendering server capable of understanding the rendering command stream, remote transmission of the rendering command stream can decouple computation from rendering, allowing the rendering cluster to be compatible with different virtualization instances and further improving resource utilization.

[0100] S25: Encapsulate the address-translated rendering instruction stream into a network packet according to a preset network protocol, and send the network packet to the rendering server so that the rendering server can execute the rendering instruction stream through the physical GPU and return the rendering result to the client for presentation.

[0101] After receiving the rendered instruction stream after address translation, the computing server can encapsulate it into network packets according to a preset network protocol through a network forwarding proxy, and then send the rendered instruction stream to the rendering server by sending network packets. This allows the rendering server to execute the rendered instruction stream through the physical GPU and return the rendering results to the client for presentation.

[0102] In practice, the network forwarding agent can standardize the rendering command stream into a network message format according to a custom network protocol. For example, it can encapsulate the address-translated rendering command stream into network packets according to the Transmission Control Protocol / Internet Protocol (TCP / IP). In other words, the network packets can support remote transmission via TCP / IP.

[0103] In this embodiment, the virtual GPU can create multiple virtual queues during initialization. These virtual queues may include a virtual instruction queue and a virtual resource queue. The virtual instruction queue stores rendering instruction streams, while the virtual resource queue stores rendering resources such as textures and buffers. After registering the virtual GPU and starting the network forwarding agent process in the user space of the computing server, the network forwarding agent can be bound to the virtual instruction queue and the virtual resource queue.

[0104] In this scenario, before sending network packets to the rendering server, the computing server can write the rendering instruction stream to the virtual instruction queue, write each rendering resource associated with the rendering instruction stream to the virtual resource queue, and trigger a notification message for the network forwarding agent.

[0105] When the computing server sends a network packet to the rendering server, it can specifically perform the following steps: after the network forwarding agent receives the notification message, it reads the rendering instruction stream and each rendering resource, encapsulates the rendering instruction stream and each rendering resource into a network packet according to a preset network protocol, and then sends the network packet to the rendering server.

[0106] Specifically, see Figure 3 This is a schematic diagram illustrating the network message sending process provided in an embodiment of this application. Taking an Android virtual container as an example, as... Figure 3As shown, the compute server can include user space and kernel space. The Android virtual container and network forwarding proxy are deployed in user space, while the VirtIO-GPU driver is deployed in kernel space. Within the Android virtual container of the compute server, when rendering for the target application, the Mesa graphics library can generate a rendering instruction stream and send the rendering instruction stream and its associated rendering resources (such as texture data) to the kernel-space VirtIO-GPU driver via the libdrm interface. The kernel-space VirtIO-GPU driver can then place the received data into two virtual queues according to data type: a virtual instruction queue and a virtual resource queue.

[0107] Meanwhile, the network forwarding proxy can use the epoll mechanism to detect file descriptors (eventfds) bound to the aforementioned virtual instruction queues and virtual resource queues in real time. Since the epoll mechanism only returns file descriptors where events have occurred, it is possible to notify the proxy by triggering a readable event for the corresponding file descriptor through eventfd_signal() when the kernel-mode VirtIO-GPU driver writes data to the two virtual queues. This allows the proxy to read the rendering instruction stream from the virtual instruction queue and the rendering resources associated with the rendering instruction stream from the four virtual resource queues.

[0108] After reading the rendering command stream and the rendering resources associated with it, the Proxy can obtain the address-translated rendering command stream through the address translation method described above. Then, it encapsulates the address-translated rendering command stream into a network packet according to a preset network protocol (e.g., TCP / IP) and sends it to the rendering server.

[0109] In the above implementation, by introducing a buffering mechanism for virtual instruction queues and virtual resource queues, and in conjunction with asynchronous retrieval by a notification message event-driven network forwarding agent, the computing logic in the virtual runtime environment can continue to proceed after the computing server writes the message into the queue, avoiding blocking and waiting for network transmission, and improving concurrent throughput.

[0110] Optionally, when the Proxy encapsulates the address-translated rendering instruction stream into network packets according to a preset network protocol, it can first serialize and reassemble the address-translated rendering instruction stream in user space, organizing the instructions and associated rendering resources that were originally scattered across multiple buffers into a continuous binary byte stream that conforms to a custom protocol. Then, it can implement a differentiated compression strategy based on the data types contained in the address-translated rendering instruction stream. For example, lossless block compression algorithms (such as LZ4 algorithm, Snappy algorithm, etc.) can be called to perform lossless compression on large blocks of resource data with high repetition such as vertex attributes and texture pixels to reduce bandwidth consumption, while the compression stage can be skipped for the rendering instruction stream to avoid consuming CPU computing power.

[0111] Optionally, the Proxy encapsulates the serialized rendering command stream and each rendering resource with multiple layers of protocol headers to obtain network packets. The inner protocol header carries the global resource identifier, physical offset, and compression flags corresponding to the address-translated rendering command stream, ensuring the receiving end can accurately reconstruct the layout of the rendering resources in video memory. The outer header contains the protocol version, packet sequence number, and packet type, used for TCP dual-channel flow control and out-of-order reordering. The packet type can include commands (cmd) and resources (res).

[0112] Optionally, the encapsulated network packets can also be sent to the rendering server via two independent TCP connections. Specifically, a command channel for sending instructions and a resource channel for sending large blocks of resources can be pre-established between the network forwarding agent and the rendering server. In this case, when the network forwarding agent encapsulates the rendering instruction stream and each rendering resource into network packets according to a preset network protocol, it can encapsulate the rendering instruction stream into instruction packets and each rendering resource into resource packets. When sending network packets to the rendering server, instruction packets can be sent to the rendering server via the command channel, and resource packets can be sent to the rendering server via the resource channel.

[0113] For example, see Figure 4 This is another schematic diagram illustrating a network message transmission process provided in an embodiment of this application. For example... Figure 4 As shown, the Proxy can pre-establish TCP / IP sockets with the remote rendering server, including a command socket: cmd_socket and a resource socket: res_socket. Command messages are passed through the command socket, and resource messages are transmitted through the resource socket.

[0114] Through the above implementation methods, the short data packets in the command messages and the stringent real-time requirements directly determine the screen response latency, while the resource messages have large single-packet sizes, numerous fragments, and long transmission times. Transmitting through two separate channels completely isolates command traffic from resource traffic, preventing them from competing for the link window and ensuring that real-time rendering commands such as mouse and viewpoint changes are delivered to the rendering server with low latency.

[0115] Furthermore, in this embodiment, the transmission between the computing server and the rendering server is optimized using a low-latency instruction stream migration protocol for real-time interaction. A high-efficiency network protocol is defined and implemented for transmitting Virgl protocol instruction streams. Compared to related technologies that transmit raw pixels or video streams, this protocol results in smaller and more compressible data volumes. Simultaneously, by combining epoll and zero-copy mechanisms, extremely low-overhead data interception and transmission can be achieved in the network forwarding proxy, and seamless integration with the rendering server's real-time rendering, hardware encoding, and WebRTC backhaul links is possible. Therefore, when applied to cloud gaming scenarios, it can meet the core requirement of ultra-low latency, and by transmitting lightweight rendering instructions instead of heavy video intermediate frames, it effectively controls network bandwidth and transmission latency, ensuring a smooth end-to-end interactive experience.

[0116] Furthermore, to further optimize network bandwidth usage, instruction stream compression technology can be introduced when transmitting network packets to the rendering server. This encodes and compresses repetitive or redundant instruction sequences, significantly reducing the amount of data transmitted per transmission. Simultaneously, in conjunction with a differential transmission mechanism, only the incremental changes between the current instruction stream and the previous baseline state are transmitted, rather than sending the complete instruction stream each time. This significantly reduces network load even in high-concurrency scenarios, ensuring the real-time performance and stability of rendering tasks.

[0117] In one optional implementation, to further improve resource utilization, the cloud server cluster may include multiple rendering servers. In this case, when sending network packets to the rendering server, the rendering server that meets the preset resource requirements may be selected from the multiple rendering servers based on the latest resource usage status of each of the multiple rendering servers, and then the network packet may be sent to the selected rendering server.

[0118] The latest resource usage status of each rendering server can be at least one status indicator describing the usage of the physical GPUs in that rendering server. Status indicators include, but are not limited to, GPU utilization, memory usage, I / O latency, and core temperature. Preset resource requirements can be the GPU resource requirements for the rendering tasks of the target application. GPU resource requirements can include, but are not limited to, memory size, the number of CUDA cores in Compute Unified Device Architecture (CUDA), and bandwidth requirements.

[0119] The selection of a rendering server that meets preset resource requirements can include, but is not limited to: selecting the rendering server with the lightest load from multiple rendering servers, while also choosing the rendering server with the highest matching degree based on the real-time load of the target application's rendering tasks, with the aim of reducing resource fragmentation. By selecting a rendering server that meets preset resource requirements, rendering tasks can be dynamically and optimally distributed to the most suitable GPU resources, thereby achieving global load balancing.

[0120] In one example, when selecting the least overloaded rendering server from multiple rendering servers to meet the resource requirements of a rendering task, the following steps can be taken: First, collect real-time GPU utilization, memory usage, core temperature, and other status metrics for each rendering server, and set one or more basic hardware conditions. For example, if the basic hardware condition is that available memory must be greater than 120% of the memory required for the rendering task, all rendering servers meeting this condition can be selected as candidate servers. Then, calculate the overall load index for each candidate server. For instance, the average GPU utilization, memory usage, and I / O latency of each candidate server can be used as evaluation metrics, and the weighted sum of the evaluation values ​​for each metric can be used to obtain the overall load index for the candidate servers. Finally, the server with the lowest overall load index can be selected as the final rendering server.

[0121] This selection strategy can effectively balance the load on each rendering server in the cloud server cluster and avoid hotspot concentration. It is particularly suitable for rendering scenarios that run continuously for a long time, because low-load rendering servers often have more computing power and heat dissipation space, which helps to improve the stability and response speed of the overall rendering task.

[0122] In another example, aiming to reduce resource fragmentation, the remaining resources of rendering servers are precisely matched based on the real-time load of the target application's rendering tasks to select the rendering server with the highest matching degree. First, an ideal resource profile can be generated based on preset resource requirements, such as the amount of VRAM, number of CUDA cores, and bandwidth requirements of the rendering tasks. Then, all rendering servers are traversed, and the currently available GPU resources of each server are quantitatively evaluated to calculate a matching degree score for each server. The matching degree score can be obtained by combining the Euclidean distance or cosine similarity between the rendering server's remaining resources and the ideal resource profile, while adding a penalty term. The penalty term can be set as follows: if the remaining resources significantly exceed the preset resource requirements, the score is lowered to avoid wasting high-configuration rendering servers executing rendering tasks with lower resource requirements; if the remaining resources are just slightly higher than the preset resource requirements, the score is increased to ensure smooth execution of rendering tasks without wasting GPU resources. Finally, the rendering server with the highest matching degree evaluation value is selected as the rendering server to issue rendering command streams. This selection strategy is suitable for hybrid rendering scenarios, and fine-grained matching can maximize the resource utilization efficiency of the overall cloud server cluster.

[0123] In addition to the selection strategies described above, other rendering server selection strategies for dynamically scheduling GPU resources can also be applied to the embodiments of this application, and will not be described in detail here.

[0124] Furthermore, based on the above selection strategies, multi-instance GPU (MIG) virtualization, priority sharding, and other strategies can be used to allocate rendering tasks on demand to further optimize resource utilization.

[0125] For example, when using the MIG virtualization strategy, a physical GPU can be divided into multiple isolated hardware instances, each with its own independent video memory, cache, and computing units, thereby achieving true hardware-level resource isolation and on-demand allocation. Across multiple rendering servers, the resource scheduler can dynamically create or adjust MIG instances based on the specific resource requirements of each rendering task (e.g., video memory capacity, number of computing cores), allocating different rendering tasks to different MIG instances for parallel execution and avoiding mutual interference.

[0126] Priority sharding is a GPU resource scheduling strategy based on time slices or weight allocation. It allows the cluster scheduling system to allocate differentiated shares of computing resources to different tasks according to their urgency or importance. Therefore, when priority sharding is used as the selection strategy, high-priority rendering tasks can obtain more time slices or higher scheduling weights, thus gaining GPU execution opportunities faster and ensuring the timely delivery of critical rendering frames; low-priority rendering tasks can be executed in idle time slices, achieving elastic utilization of resources and avoiding idle waste.

[0127] In some embodiments, each of the plurality of rendering servers can be a rendering server dedicated to performing rendering tasks, or it can be a rendering server used to perform both computational and rendering tasks. When a rendering server is dedicated to performing rendering tasks, the rendering tasks it performs can be only rendering tasks for the target application, or they can be rendering tasks for multiple applications. When a rendering server is used to perform both computational and rendering tasks, the computational tasks it performs can be computational tasks for other applications, or they can be computational tasks for other platform versions of the target application.

[0128] For example, taking Android version application A as the target application, the cloud server cluster can include rendering servers 1 to 10. Rendering servers 1 to 3 can be servers dedicated to executing rendering tasks for application A. Rendering servers 4 and 5 can be servers used to execute multiple applications, including application A. Rendering servers 6 and 7 can be servers used to execute computing and rendering tasks for Windows version application A. Rendering servers 8 to 10 can be servers used to execute computing and rendering tasks for application B.

[0129] In this embodiment, the technical feature of using a virtual GPU in a virtual runtime environment instead of a physical GPU in the computing server, and sending the rendering instruction stream to the physical GPU on a remote rendering server for execution, achieves physical decoupling between computing and rendering. This allows computing and rendering resources to be planned and scaled elastically independently, avoiding the resource contention or idleness issues found in traditional all-in-one servers, and significantly improving overall resource utilization. Furthermore, faults between the computing server and the rendering server are isolated from each other; a single resource layer failure does not affect the operation of other service layers, thereby improving the overall robustness and availability of the system.

[0130] Furthermore, by converting graphics interface call instructions into a rendering instruction stream that is common across multiple hardware platforms, transparent compatibility with different underlying GPU hardware can be achieved. Rendering can be completed on any physical GPU model without modifying the graphics framework or driver of the target application, ensuring broad ecosystem compatibility.

[0131] See Figure 5 The diagram shown is a flowchart of an implementation of a cloud rendering method provided in this application. The execution entity of this method is a rendering server in a cloud server cluster. The specific implementation process of this method is as follows: S51~S54: S51: Receive network packets sent by the computing server and obtain the rendering instruction stream by parsing the network packets; wherein, the network packets are generated in the following way: the computing server performs address translation on the rendering instruction stream through a deployed network forwarding proxy, and then encapsulates the address-translated rendering instruction stream according to a preset network protocol to obtain network packets, so as to convert the virtual address information contained in the rendering instruction stream into physical offsets that the rendering server can recognize; the virtual address information is used to indicate the addresses of each rendering resource associated with the rendering instruction stream in the virtual runtime environment; the rendering instruction stream is generated in the following way: in the virtual runtime environment, the computing server runs the target application according to the operation instructions triggered by the client for the target application, obtains the running results, and generates graphics interface call instructions corresponding to the running results for the virtual GPU, and converts the graphics interface call instructions into the rendering instruction stream; the instruction format of the rendering instruction stream is a common instruction format across multiple hardware platforms.

[0132] For details on how network packets and rendering command streams are generated, please refer to [link / reference]. Figure 2 The relevant descriptions in the method embodiments shown will not be repeated here.

[0133] Specifically, the rendering server can continuously receive network packets through the communication channel with the computing server, and then extract the payload from the network packets and deserialize them. That is, according to the encoding rules agreed upon by the computing server, the network is restored into a structured object, thereby obtaining the original, platform-independent rendering instruction stream.

[0134] In one optional implementation, when a command channel and a resource channel are pre-established between the network forwarding proxy of the computing server and the rendering server, the rendering server can receive command packets from network packets through the command channel and resource packets from network packets through the resource channel. Then, the command packets can be parsed to obtain the rendering command stream, and the resource packets can be parsed to obtain the rendering resources associated with the rendering command stream.

[0135] Specifically, when the rendering server establishes a connection with the network forwarding proxy of the computing server, and pre-creates an instruction channel and a resource channel, the rendering server can continuously receive instruction messages from the computing server through the instruction channel. Upon receiving a complete instruction message, it extracts the payload, deserializes it according to the encoding rules agreed upon by both parties, and restores it to a platform-independent structured object, thus obtaining the original rendering instruction stream. This stream may contain, but is not limited to, scheduling information such as task identifiers, scene references, frame parameters, and output paths. Simultaneously, the rendering server can also continuously receive resource messages through the resource channel. The payloads of these messages carry various rendering resources (such as geometric models, texture maps, and shader code) referenced by the rendering instruction stream. After deserialization, the resource data is written to the local cache or shared storage for subsequent rendering processes to load on demand. The two channels operate independently and do not block each other, allowing instruction issuance and resource transmission to proceed in parallel. This avoids the protocol complexity and parsing latency caused by mixing instructions and resources in the same channel, and ensures that high-bandwidth resource transmission does not crowd out the performance of low-latency instruction response channels.

[0136] Before executing S51, the rendering server can perform service initialization. Specifically, the rendering server can start a daemon process to detect connection requests from the compute server in real time. When a connection request is detected, one or more channels for transmitting messages can be established with the corresponding compute server, thus establishing a new connection. After establishing a new connection, an independent child process can be created for each new connection to handle it and maintain command sockets (command channels) and resource sockets (resource channels). Each child process can continuously receive corresponding messages from the established channels, parse the messages, and then hand over the parsed rendering tasks to the rendering pipeline for execution.

[0137] Then, the graphics environment can be initialized. Specifically, the rendering server can connect to a local display compositor (such as Wayland) to initialize the EGL display, context, and Virgl renderer, preparing a complete local graphics execution environment for subsequent rendering tasks.

[0138] After service initialization and graphics environment initialization are completed, the instruction subprocess can continuously receive data packets from the instruction socket, and the resource subprocess can continuously receive data packets from the resource socket. The data packets are then decapsulated and parsed to reconstruct the original rendering instruction stream and various rendering resources.

[0139] S52: Decode the rendering instruction stream to obtain executable drawing instructions.

[0140] In practice, the rendering server can submit the restored rendering instruction stream and various rendering resources to the rendering service library within the rendering server. The rendering service library, acting as a host-side instruction translation library, can decode the rendering instruction stream into standard, executable drawing instructions recognizable by the rendering server's operating system, according to the VirGL protocol. These executable drawing instructions can be used with… Figure 2 The graphics interface call instructions (such as OpenGL ES calls) shown are the same. For example, the rendering service library can be a Virgl Renderer library initialized locally on the rendering server.

[0141] In some embodiments, in order for the rendering server to convert the rendering instruction stream into executable drawing instructions, the rendering server typically needs to include a Virgl graphics technology stack, which may include the following four components: (1) a VirtIO-GPU driver; (2) a Mesa graphics library containing a Virgl driver; (3) a VirtIO-GPU virtual graphics card implemented on the Qemu side; and (4) a Virgl Renderer library on the host side.

[0142] S53: Executes executable drawing instructions through the physical GPU to obtain rendering results.

[0143] In one alternative implementation, after parsing out each rendering resource associated with the rendering instruction stream, the rendering result can be obtained by executing executable drawing instructions through the physical GPU based on the obtained rendering resources.

[0144] In practice, the executable drawing instructions restored by S52, such as OpenGL calls or OpenGL ES calls, can be passed to the Mesa graphics library or other graphics drivers of the rendering server. Then, the physical GPU driver takes over and drives the physical GPU hardware to complete the actual rendering task and obtain the rendering result.

[0145] Taking OpenGL calls as the executable drawing instructions and the Mesa graphics library as the graphics driver as an example, as a graphics driver framework, the Mesa graphics library first performs state checks and verifications on the OpenGL calls. Specifically, it can maintain an internal OpenGL state machine to check whether the current context is valid, whether the texture / buffer object exists, and whether the shader has been compiled and linked, etc.

[0146] OpenGL calls can include drawing class calls and state modification class calls. Drawing class calls are those that directly trigger the physical GPU to perform rendering operations, such as glDrawArrays and glDrawElements; state modification class calls are those used to change the OpenGL state machine records maintained internally by the Mesa graphics library, and do not directly trigger hardware operations, such as binding textures, modifying state switches, and setting shader parameters.

[0147] In this case, after the state verification is passed, the Mesa graphics library can record these non-drawing class calls as a set of current states of the internal driver. When a drawing class call is encountered, the driver can translate all the currently accumulated states, together with the entry address of the previously uploaded shader binary code, the location information of each rendering resource in the video memory, and other related content, into a series of configuration instructions and startup instructions that the physical GPU can recognize, and write them sequentially into a memory area called the command buffer.

[0148] The location information of each rendering resource in video memory is determined as follows: For each rendering resource, the rendering server can first determine its physical storage start address in the rendering server based on the global resource identifier of the rendering resource, and then calculate its read position by combining the physical offset of the rendering resource, and use the determined read position as the location information of the rendering resource in video memory.

[0149] For example, suppose the Mesa graphics library needs to read a specific texture resource from the rendering resources for sampling when processing a drawing class call (such as glDrawElements). The Mesa library first obtains the global resource identifier (e.g., TEX_ID_001) of the texture resource from the current state machine, and then locates the starting address of the texture's physical storage in the rendering server's physical video memory (e.g., 0x7F800000) by querying the resource mapping table maintained locally by the rendering server. Subsequently, the Mesa library obtains the physical offset carried in the instruction (e.g., 0x00002000). It should be noted that this physical offset is a virtual offset from the virtual runtime environment, translated and included in the network protocol. The Mesa library adds the physical storage starting address and the physical offset together (i.e., 0x7F800000 + 0x00002000 = 0x7F802000) to accurately determine the actual read location of the texture data in the physical video memory. The Mesa graphics library then uses the final physical address (0x7F802000) as part of the configuration instructions, that is, as the location information of the texture resource in video memory, to write it into the command buffer.

[0150] Subsequently, the Mesa graphics library can submit this command buffer to the physical GPU's scheduling queue through the display driver interface provided by the operating system kernel. The kernel-level physical GPU driver can then send it to the hardware front-end engine, where the physical GPU hardware parses the instructions in the command buffer line by line, and finally writes the rendering result to the frame buffer or the video memory area where the specified off-screen rendering target is located through the rendering output unit. At this point, the rendering result of a frame of the target application is stored in video memory, ready for subsequent reading or transmission.

[0151] S54: Return the rendering result to the client so that the client can display the rendering result.

[0152] In practical implementation, the rendered result, i.e., the rendered frame, can be directly hardware encoded by the rendering server with low latency. The encoding standard can be H.264 or H.265, etc., and this application does not impose specific restrictions on it. The encoded video stream can be adaptively transmitted to the client through another low-latency channel (such as the RTC protocol channel: WebRTC), supporting real-time interaction in weak network environments.

[0153] Based on the same inventive concept, embodiments of this application also provide a cloud rendering system. This cloud rendering system may include a client, a computing server, and a rendering server, wherein the computing server is deployed with a virtual runtime environment containing a virtual GPU, and the rendering server is deployed with a physical GPU.

[0154] The client is the user terminal, used to respond to the user's operation on the target application, trigger operation instructions, and send the operation instructions to the computing server.

[0155] The computing server receives operation instructions, runs the target application according to the operation instructions in the virtual runtime environment, obtains the running results, generates graphics interface call instructions corresponding to the running results for the virtual GPU, converts the graphics interface call instructions into a rendering instruction stream, and performs address translation on the rendering instruction stream through a network forwarding proxy to convert the virtual address information contained in the rendering instruction stream into physical offsets that the rendering server can recognize; the virtual address information is used to indicate the addresses of each rendering resource associated with the rendering instruction stream in the virtual runtime environment; the address-translated rendering instruction stream is encapsulated into network packets according to a preset network protocol and sent to the rendering server.

[0156] The rendering server receives network packets sent by the computing server and obtains the rendering instruction stream by parsing the network packets; it decodes the received rendering instruction stream to obtain executable drawing instructions; it executes the executable drawing instructions through the physical GPU to obtain the rendering result; and it returns the rendering result to the client.

[0157] The client, after receiving the rendering result, also uses it to present the rendering result in the display interface.

[0158] The specific implementation methods of the computing server and rendering server can be found in the relevant descriptions in the above method embodiments, and will not be repeated here.

[0159] See Figure 6 This is a schematic diagram of a cloud rendering system provided in an embodiment of this application. Figure 6 As shown, a cloud rendering system can include a client and a server. The client can be a lightweight terminal device (tiny-client). The server can include a compute server and a rendering server. Users can operate the application on the terminal device that launches the cloud rendering method for the target application, and upload operation instructions for the target application to the compute server. For example, a user can open the cloud application version of the target application in a browser and perform a series of operations on the target application in the cloud application version, such as clicking, dragging, etc.

[0160] The computing server can be a server containing only CPUs and no physical GPUs. After receiving operation instructions uploaded by the client, it can run the target application in a virtual runtime environment according to the operation instructions, generate a rendering instruction stream, and send it to the rendering server.

[0161] After receiving the rendering instruction stream, the rendering server can decode it into executable drawing instructions that can be recognized by the deployed physical GPU, and use the physical GPU to perform accelerated rendering to obtain the rendering result, which is then fed back to the client via RTC.

[0162] It should be noted that the number of computing servers and rendering servers is not specifically limited in this embodiment.

[0163] In one optional implementation, the cloud rendering system may include multiple rendering servers, and may also include a resource scheduler. The resource scheduler can dynamically allocate GPU resources from the multiple rendering servers according to the rendering load of the rendering task. Specifically, the resource scheduler can be used to select a rendering server that meets preset resource requirements from the multiple rendering servers based on the latest resource usage status of each of the multiple rendering servers, so that the computing server can send network packets.

[0164] Optionally, the resource scheduler can be deployed on a computing server or a dedicated scheduling server in the cloud rendering system to act as a resource scheduler, distributing rendering tasks to selected rendering servers based on the GPU resource requirements of the rendering tasks and the resource utilization of the rendering servers.

[0165] Taking a cloud rendering system with a dedicated scheduling server as the resource scheduler, and assuming the selection strategy is to choose the least loaded rendering server from multiple rendering servers that meets the resource requirements of the rendering task, as an example, see [link to example]. Figure 7 This is a schematic diagram of dynamic resource scheduling provided in an embodiment of this application. Assume the resource requirements of the rendering task are: at least approximately 3GB of video memory is needed to accommodate scene geometry or texture uploading and frame buffering. Therefore, rendering servers with available video memory ≥ 3072MB × 120% = 3686MB are selected as candidate servers from rendering servers 1 to n. Assume the selected candidate servers are rendering servers 1 to 3.

[0166] Then, further selection can be made based on the real-time GPU status of rendering servers 1 to 3. The real-time GPU status of rendering server 1 is 11GB of available video memory, with an average GPU utilization of 40%, a video memory usage of 54%, and an IO wait time of 0.2 (normalized to a relative value of 0 to 1); the real-time GPU status of rendering server 2 is 6GB of available video memory, with an average GPU utilization of 78%, a video memory usage of 75%, and an IO wait time of 0.6; the real-time GPU status of rendering server 3 is 8GB of available video memory, with an average GPU utilization of 65%, a video memory usage of 50%, and an IO wait time of 0.45.

[0167] Assuming the overall load index is calculated using a weighted average of GPU utilization, memory usage, and I / O wait time, with GPU utilization having a weight of 0.5, memory usage having a weight of 0.3, and I / O wait time having a weight of 0.2, then the overall load index for rendering server 1 is: 0.5×0.4+0.3×0.54+0.2×0.2=0.2+0.162+0.04=0.402; the overall load index for rendering server 2 is: 0.5×0.78+0.3×0.75+0.2×0.6=0.39+0.225+0.12=0.735; and the overall load index for rendering server 3 is: 0.5×0.65+0.3×0.5+0.2×0.45=0.325+0.15+0.09=0.565.

[0168] Since 0.402 < 0.565 < 0.735, rendering server 1 can be selected as the rendering server that issues rendering command streams.

[0169] In conjunction with the cloud rendering system described above, see [link to relevant documentation]. Figure 8 This is a schematic diagram of the overall architecture of the cloud rendering method provided in the embodiments of this application. Figure 8 Taking the target application as an Android application as an example, such as Figure 8As shown, user-triggered operation commands for the target application via the client serve as input to the computing server. For audio, the computing server performs audio-related processing and then feeds the results back to the client. For image rendering in video, the computing server runs the target application within an Android virtual container and generates a rendering command stream using the Mesa graphics library. This stream is then sent to the rendering server via the VirtIO-GPU driver and the VirtIO-GPU virtual graphics card. During transmission, a network forwarding proxy can be used first. Figure 8 (Not shown in the image) performs address translation and packet encapsulation according to the TCP network protocol, and then sends the encapsulated network packet to the rendering server.

[0170] After receiving a network packet, the rendering server can use the Virgl Renderer library to decode the rendering command stream and drive the physical GPU through the Mesa graphics library to execute the decoded executable drawing commands, thus obtaining the rendering result. Based on the rendering result, the rendering server can perform hardware encoding on the video stream and send the encoded video stream back to the client.

[0171] The above implementation method enables heterogeneous migration of rendering tasks among multiple cloud rendering servers based on network transmission, thereby facilitating low-cost and efficient graphics computing and ultimately improving the utilization rate of GPU resources. The centralized resource scheduler can dynamically and optimally distribute rendering tasks to the most suitable GPU resources according to the real-time load of different services, achieving global load balancing across services and clusters.

[0172] based on Figure 8 The overall architecture shown is available for reference. Figure 9 The diagram shown is a schematic representation of the overall process of a cloud rendering method provided in an embodiment of this application. The specific implementation process of this method is as follows: S901: Perform the operation.

[0173] Users can perform corresponding operations on the target application on the client, such as clicking and touching.

[0174] S902: The client inputs operation commands to the computing server.

[0175] The client can serialize the user's corresponding operations into operation instructions and send them to the computing server.

[0176] S903: The computing server executes the computing logic of the target application through a virtualized runtime environment, obtaining virgl protocol instruction streams and audio streams.

[0177] The computing server can execute the computing logic and audio processing of the target application through a virtualized runtime environment, and obtain platform-independent virgl protocol instruction streams and processed audio streams.

[0178] S904: The computing server outputs the processed audio stream to the client.

[0179] Regarding the timing of S904's execution in this process, that is, S904 can be executed at any time before S911, this application does not impose specific restrictions.

[0180] S905: The compute server sends the virgl protocol instruction stream to the VirtIO-GPU driver through the libdrm interface.

[0181] S906: The compute server writes virgl protocol instruction streams and rendering resources to a virtual queue via the VirtIO-GPU driver.

[0182] S907: The compute server sends the virgl protocol command stream and rendering resources to the rendering server through a network forwarding agent.

[0183] S908: The rendering server parses the Virgl protocol instruction stream through the Virgl Renderer.

[0184] S909: The rendering server drives the physical GPU via Mesa / GPU driver.

[0185] The rendering server drives the physical GPU through the Mesa graphics library or a GPU driver.

[0186] S910: The rendering server performs rendering operations through the physical GPU to obtain the rendering results.

[0187] S911: The rendering server outputs the encoded video stream to the client.

[0188] S912: The client decodes and displays the rendering result.

[0189] The client can decode the encoded video stream, combine it with the audio stream to perform audio-visual synchronization, and then display the rendering result on the interface.

[0190] For detailed implementation methods of the above steps, please refer to [link / reference]. Figure 2 or Figure 5 The relevant descriptions in the method embodiments shown will not be repeated here.

[0191] The cloud rendering method provided in this application embodiment will be introduced below in conjunction with the overall implementation process described above, for two specific application scenarios.

[0192] For example, in remote hardware rendering acceleration scenarios in cloud gaming, cloud phones, or other cloud applications, the cloud rendering method provided in this application embodiment can support users to smoothly run high-performance graphics applications and large games on consumer-grade terminal devices without GPU configurations (such as smartphones, tablets, and thin and light laptops). Alternatively, it can provide application services with high graphics processing requirements on low-computing-power terminal devices, such as Augmented Reality (AR) / Virtual Reality (VR) applications, high-definition video editing, and 3D design and modeling software.

[0193] See Figure 10 This is a schematic diagram of cloud rendering methods in related technologies. For example... Figure 10 As shown, taking cloud gaming as an example, cloud gaming servers in related technologies typically deploy CPUs and GPUs on the same machine. However, this model is a monolithic server model with strong coupling between computing and rendering, and it has the following inherent drawbacks: (1) Resource rigidity and poor scalability: Each server must be equipped with both CPU and GPU, and GPU resources are physically bound to virtual machine instances. Independent and elastic scaling based on the different resource requirements of different task types is not possible. For example, compute-intensive tasks (such as game logic) and rendering-intensive tasks have different requirements for GPU and CPU resources. In this case, if you want to expand the GPU capacity of the cloud server cluster for rendering-intensive tasks, you must also add CPU servers, and vice versa, resulting in low resource utilization and inflexible cluster expansion.

[0194] (2) Heavy hardware dependence and high cost: In order to achieve the best compatibility and performance, cloud servers usually need to adopt the ARM architecture to natively match the Android system, resulting in a narrow range of server hardware choices and high procurement and maintenance costs. At the same time, it is impossible to utilize heterogeneous (such as x86) or more efficient GPU computing resources in the data center.

[0195] (3) Unable to fully utilize GPU cluster performance: GPUs are fixed within a single server, making it impossible to form a shared, on-demand, large-scale rendering computing power pool. It is difficult to achieve load balancing and redundancy backup across multiple servers, which limits the upper limit and reliability of the overall graphics processing capability of the cloud server cluster.

[0196] (4) Complex device adaptation: Since it directly depends on the local GPU driver of the computing server, any change in GPU hardware or driver upgrade may require adjustments to the entire Android virtualization image or streaming service, resulting in high operational complexity and device fragmentation issues.

[0197] Therefore, currently Figure 10The cloud rendering solutions shown face fundamental bottlenecks in pursuing higher density, lower cost, and more flexible cloud graphics services. To address this, this application provides a remote rendering acceleration technology that separates rendering from computation and decouples heterogeneous resources.

[0198] Specifically, the cloud rendering method provided in this application can solve the defects in the above-mentioned related technologies from three levels: architecture, protocol, and compatibility.

[0199] In terms of architecture, the above embodiment achieves physical decoupling of the three-stage pipeline of computation, rendering, and encoding. Specifically, a network forwarding proxy is inserted at a key node in the data path of the Virgl graphics technology stack (Virtio-GPU driver layer). This proxy intercepts the platform-independent Virgl rendering instruction stream that should be processed by the local Virgl Renderer and transmits it to a remote independent GPU rendering cluster (a cluster of rendering servers) via a low-latency network. This completely breaks the tight coupling mode of co-deploying computation and rendering. It achieves resource pooling and elastic scaling, that is, GPU resources are centralized into an independent cluster, which can be shared on demand by all compute servers without GPUs, greatly improving resource utilization and significantly reducing hardware costs. It also enables fault isolation, that is, the failure of compute nodes and rendering nodes does not affect each other, enhancing system availability. Moreover, when the physical GPU on a rendering server fails (or is overloaded), its rendering tasks can be taken over by the physical GPUs of other rendering servers, ensuring uninterrupted service and achieving load balancing and redundancy backup across multiple servers. It solves the defects of resource rigidity, poor scalability, and inability to fully utilize the performance of GPU clusters in related technologies.

[0200] In terms of transmission, the above embodiments provide a low-latency command stream migration protocol for real-time interaction. Specifically, this application defines and implements a high-efficiency network protocol for transmitting Virgl protocol command streams. Compared to transmitting raw pixels or video streams, this protocol has a smaller data volume and is compressible. Simultaneously, by combining epoll, zero-copy, and other mechanisms, it achieves extremely low-overhead data interception and transmission in the forwarding proxy, and seamlessly integrates with the receiving end's real-time rendering, hardware encoding, and WebRTC backhaul link. This meets the core requirement of cloud gaming for ultra-low latency. Moreover, by transmitting lightweight rendering commands instead of heavy video intermediate frames, network bandwidth and transmission latency can be effectively controlled, ensuring a smooth end-to-end interactive experience. Through improvements at the transmission level, the aforementioned shortcomings in related technologies can be resolved while avoiding other defects such as excessive latency.

[0201] In terms of compatibility, the above embodiments can seamlessly adapt to the containerization and Android virtualization ecosystems. Specifically, this application embodiment delves into the architecture where containers and host machines share the same kernel. By intercepting and redirecting ioctl calls sent by applications within the container to the kernel Virtio-GPU driver via libdrm, transparent capture of rendering tasks is achieved. This is completely unaffected by the applications or graphics framework (SurfaceFlinger) within the Android virtual container, requiring no modifications. Therefore, it perfectly adapts to the cloud mobile game deployment model of the containerization era. It provides remote 3D graphics acceleration capabilities, identical to a local GPU, for high-density Android containers running on GPU-less servers, solving the problem that virtual machine solutions in related technologies are unsuitable for container environments. Furthermore, regarding the complexity of adapting to the aforementioned device models, since the rendering server executes rendering tasks by parsing rendering instruction streams, and these rendering instruction streams are platform-independent (i.e., independent of the hardware configuration of the rendering server), even if the GPU hardware changes or the driver is upgraded, there is no need to modify the entire Android virtualization image or streaming service in the computing server, thereby reducing operational complexity and solving the problem of device fragmentation.

[0202] Furthermore, regarding platform abstraction, this application's embodiments achieve cross-instruction set compatibility based on the Virgl protocol. Specifically, the Virgl instruction stream itself is a hardware-independent intermediate representation of rendering commands. This application's embodiments leverage this characteristic to allow rendering tasks running on ARM architecture containers to be seamlessly migrated to x86 architecture GPU servers. This achieves true heterogeneous hardware hybrid deployment. Service providers can flexibly combine ARM computing clusters and x86 GPU rendering clusters based on cost-effectiveness, breaking through hardware architecture limitations, optimizing overall costs, and solving the shortcomings of related technologies such as severe hardware dependence and high costs.

[0203] Therefore, this application embodiment, by creatively migrating the Virgl rendering instruction stream over a network, not only solves the fundamental defects of existing solutions in terms of resource utilization, latency, compatibility, and heterogeneous support at the technical level, but also provides a highly cost-effective, highly elastic, and scalable innovative architecture for cloud gaming and cloud phone services at the commercial level, representing an important direction in the evolution of cloud graphics computing.

[0204] The following examples, through two different application scenarios, further illustrate the cloud rendering method provided by this application.

[0205] See Figure 11This is a schematic diagram illustrating the rendering process of a cloud rendering method in a given scenario, as provided in the application embodiment. Taking a cloud gaming scenario as an example, this method employs a distributed architecture that separates computation and rendering physics, replacing the tightly coupled model of CPU and GPU co-deployment in related cloud gaming technologies. The system consists of the following three core parts: Client: In this scenario, assuming the target application is game aaa, the client is used to receive user input for game aaa, decode it, and present the audio and video streams.

[0206] Cloud gaming server A (a computing node equipped with only CPU): used to run the Android virtual runtime environment, process the running logic, operation instructions and audio of game AAA, and generate rendering instruction streams. Figure 11 The AIC included refers to the audio codec chip used to perform audio processing.

[0207] Cloud gaming server B (containing rendering nodes with CPU and physical GPU): It is used to receive rendering instruction streams from cloud gaming server A, complete graphics rendering, video encoding, and output video streams.

[0208] The overall rendering process includes: Users interact with the game via a lightweight client (e.g., touch, click), and the client encodes these interactions into control commands, which are then uploaded to cloud gaming server A via a low-latency channel. Cloud gaming server A runs the game 'aaa' within an Android virtual container, processes the received control commands, and performs game logic calculations. The graphics engine generates corresponding OpenGL ES calls based on the logic state. The Mesa graphics library (integrated with a Virgl driver) within the Android virtual container encapsulates OpenGL ES calls into platform-independent Virgl protocol command streams. These encapsulated Virgl protocol command streams are transmitted to the host machine, i.e., cloud gaming server A, via the VirtIO-GPU channel. The host machine then transmits network packets containing these command streams across nodes to cloud gaming server B via a network protocol (e.g., TCP).

[0209] Cloud gaming server B receives the Virgl protocol instruction stream, parses it using the Virgl Renderer component, and restores it to standard OpenGL ES calls. The physical GPU performs hardware-accelerated rendering, generating render frames (i.e., the rendering result). The render frames are compressed into a video stream by the GPU hardware encoder (such as an encoder supporting H.264 / H.265). Cloud gaming server B can then send the encoded video stream back to the user terminal via a real-time transmission protocol (such as WebRTC). The client can decode the video stream and combine it with the audio stream from cloud gaming server A to achieve audio-visual synchronization and final presentation.

[0210] Through the above embodiments, since the computing servers do not need to be equipped with GPUs, high-density deployment can be achieved using general-purpose servers, significantly reducing the cost per server. GPU resources can also be decoupled from traditional monolithic servers and centrally built into a scalable and shareable rendering resource pool, supporting multi-tenancy and multi-service sharing, greatly improving the overall utilization rate of GPUs. This reduces overall costs and significantly improves resource utilization.

[0211] Furthermore, computing resources (CPU / memory) and rendering resources (GPU) can be independently scaled up and down according to business load, easily handling sudden traffic surges and business growth, effectively optimizing resource allocation flexibility. This achieves physical and logical decoupling of computing and rendering. Computing server and rendering server failures are mutually isolated; a single resource layer failure does not affect the operation of other service layers. Combined with cluster high availability mechanisms, this improves overall system availability. When a physical GPU on a rendering server fails (or is overloaded), its rendering tasks can be taken over by physical GPUs on other rendering servers, ensuring uninterrupted business operations and achieving redundancy backup.

[0212] In addition, service providers can independently plan and deploy computing and rendering clusters based on factors such as data center power supply, heat dissipation, network and regional costs, breaking through the physical limitations of traditional homogeneous deployment and achieving optimal energy efficiency and cost layout.

[0213] In another application scenario, the cloud rendering method provided in this application embodiment can also be applied to the hybrid deployment of different technology stacks and hardware architectures to achieve cross-instruction set rendering load balancing.

[0214] Taking the hybrid deployment of cloud-based mobile gaming and cloud gaming as an example, the current cloud gaming business mainly presents two mainstream forms, corresponding to two different technology stacks and hardware architectures: cloud gaming mainly uses the ARM architecture and relies on containerization technologies (such as Docker) to virtualize the Android system and game applications. Its advantages are high deployment density and low cost per instance. Cloud gaming mainly uses the x86 architecture and usually provides a Windows gaming environment through virtual machines or sandboxing technology, which can bring higher definition graphics and a better operating experience, but requires significantly higher computing power costs.

[0215] With the increasing popularity of high-quality cross-platform games, many cloud gaming service providers need to operate both cloud mobile games and cloud mobile games simultaneously. However, due to the fundamental differences in their underlying instruction sets (ARM and x86) and virtualization technology stacks, they are typically deployed in isolation in traditional solutions, forming independent resource pools. This not only causes expensive GPU resources to become isolated between different services, resulting in severe resource fragmentation and idle waste, but also makes it difficult to further improve overall resource utilization and cope with dynamically changing mixed loads.

[0216] See Figure 12This diagram illustrates a comparison of idle computing power fragments in cloud-based mobile games and cloud gaming within related technologies. (Example:) Figure 12 As shown, with a fixed total computing power of physical GPUs, cloud gaming has a relatively low single-task load. This allows a single server to deploy multiple mobile game instances on the same GPU using containerization technology, or to run some lightweight graphics tasks interspersed, effectively filling the small amount of remaining computing power and maintaining a high overall utilization rate. However, PC games have extremely high and continuous high-pressure requirements for single-task computing power of physical GPUs. A single PC game instance often fills most of the GPU's resources. This extremely high resource exclusivity leads to a severe resource isolation effect, making the remaining fragmented computing power insufficient to support the scattered computing power of another PC game launch, forcing it to remain idle. Therefore, cloud gaming typically has more idle computing power fragments than cloud mobile gaming, making the computing power cost required for cloud gaming higher. Furthermore, due to the different instruction sets, related technologies require complex conversions between instruction sets, which not only increases additional overhead but may also lead to compatibility issues. Therefore, the hybrid deployment of cloud mobile games and cloud gaming has been slow to materialize.

[0217] However, the cloud rendering method provided in this application allows the rendering instruction stream to be migrated over the network to a suitable rendering server for processing. This technology can ignore differences in instruction sets, enabling cloud mobile games and cloud-based games to coexist on the same platform and dynamically allocate resources according to the actual load, thereby achieving more efficient resource utilization.

[0218] See Figure 13 This is a schematic diagram illustrating the rendering process of a cloud rendering method in another scenario provided in the application embodiment. For example... Figure 13 As shown, a cloud gaming server can function as a computing server, running an Android virtual environment to execute the game's logic and process audio accordingly. For cloud gaming servers with a significant amount of idle computing power, they can act as rendering servers, utilizing the idle computing power of cloud gaming servers to perform rendering tasks for cloud gaming, thus achieving a hybrid deployment of cloud gaming and cloud-based gaming.

[0219] In addition, a heterogeneous resource scheduling platform can be added between cloud mobile game servers and cloud-based mobile game servers. This platform can dynamically and optimally distribute rendering tasks to the most suitable rendering servers based on the real-time load of different services such as cloud mobile games and cloud-based mobile games, achieving global load balancing across services and clusters. The scheduling method of the heterogeneous resource scheduling platform can be found in the description of the resource scheduler mentioned above, and will not be repeated here. Through a unified heterogeneous resource scheduling platform, logical pooling and dynamic sharing of GPU resources between cloud mobile games and cloud-based mobile games can be achieved, effectively solving the problem of computing power fragmentation caused by differences in business peak and valley periods and architectural isolation. Furthermore, it allows for flexible combination of ARM architecture computing servers and x86 architecture rendering servers according to business characteristics and the principle of optimal economics, achieving overall total cost of ownership optimization.

[0220] In addition, in this embodiment of the application, GPU rendering tasks can also be dynamically migrated between different rendering servers within the cloud server cluster, realizing automatic fault switching, resource hotspot dispersal and seamless hardware maintenance, thus ensuring service continuity.

[0221] Based on the same inventive concept, embodiments of this application also provide a cloud rendering device. For example... Figure 14 The diagram shows a schematic of a cloud rendering device 1400, applied to a computing server in a cloud server cluster. The computing server deploys a network forwarding proxy and a virtual runtime environment containing a virtual GPU. The cloud server cluster also includes a rendering server for executing rendering tasks, and the rendering server deploys a physical GPU. It may include: The computing unit 1401 is configured to receive operation instructions triggered by the client for a target application, run the target application according to the operation instructions in the virtual runtime environment, and obtain the running results; generate graphics interface call instructions corresponding to the running results for the virtual GPU; and convert the graphics interface call instructions into a rendering instruction stream; wherein the instruction format of the rendering instruction stream is a common instruction format between multiple hardware platforms. The communication unit 1402 is configured to perform address translation on the rendering instruction stream through the network forwarding proxy, so as to convert the virtual address information contained in the rendering instruction stream into a physical offset that can be recognized by the rendering server; the virtual address information is used to indicate the address of each rendering resource associated with the rendering instruction stream in the virtual runtime environment; the address-translated rendering instruction stream is encapsulated into a network packet according to a preset network protocol, and the network packet is sent to the rendering server, so that the rendering server executes the rendering instruction stream through the physical GPU and returns the rendering result to the client for presentation.

[0222] Optionally, the communication unit 1402 is specifically used to: call an address mapping function to convert the virtual address information contained in the rendering instruction stream into an address pointer accessible by the network forwarding agent; The rendering instruction stream is read through the address pointer, and the virtual resource identifier and virtual offset of each rendering resource associated with the rendering instruction stream are extracted; the virtual resource identifier is the resource identifier in the virtual runtime environment, and the virtual offset represents the offset of the corresponding rendering resource based on the resource start address in the virtual runtime environment; Based on the virtual resource identifier and virtual offset of each rendering resource, the global resource identifier and physical offset of each rendering resource are determined; the physical offset represents the offset of the corresponding rendering resource from the physical storage start address in the rendering server; the physical storage start address is located based on the global resource identifier of the corresponding rendering resource. Based on the global resource identifier and physical offset of each rendering resource, the rendering instruction stream after address translation is obtained.

[0223] Optionally, the network forwarding proxy is pre-bound to the virtual instruction queue and virtual resource queue created by the virtual GPU; before sending the network packet to the rendering server, the communication unit 1402 is further configured to: Write the rendering instruction stream into the virtual instruction queue, write each rendering resource associated with the rendering instruction stream into the virtual resource queue, and trigger a notification message for the network forwarding proxy; When sending the network message to the rendering server, the communication unit 1402 is specifically used for: After receiving the notification message, the network forwarding agent reads the rendering instruction stream and each rendering resource, and encapsulates the rendering instruction stream and each rendering resource into a network packet according to a preset network protocol; The network message is sent to the rendering server.

[0224] The network message includes command messages and resource messages; the communication unit 1402 is specifically used for: The rendering instruction stream is encapsulated into instruction packets according to a preset network protocol, and each rendering resource is encapsulated into a resource packet according to a preset network protocol. The communication unit 1402 is specifically used for: The instruction message is sent to the rendering server via the instruction channel; The resource message is sent to the rendering server through the resource channel; The instruction channel and the resource channel are pre-established communication channels between the network forwarding agent and the rendering server.

[0225] Optionally, the cloud server cluster includes multiple rendering servers, and the communication unit 1402 is specifically used to: select a rendering server that meets the preset resource requirements from the multiple rendering servers based on the latest resource usage status of each of the multiple rendering servers. The network message is sent to the selected rendering server.

[0226] Optionally, before receiving an operation instruction triggered by the client for the target application, the communication unit 1402 is further configured to: Based on the graphics attributes of each application in the virtual runtime environment, including the target application, determine the set of virtual GPU functions that need to be deployed; Based on the aforementioned feature set, a virtual GPU is registered in the kernel of the computing server, and the virtual GPU is initialized. The initialized virtual GPU is mounted into the virtual runtime environment.

[0227] Based on the same inventive concept, embodiments of this application also provide a cloud rendering device. For example... Figure 15 The diagram shows a schematic of a cloud rendering device 1500, used as a rendering server in a cloud server cluster. The rendering server is equipped with a physical GPU. The cloud server cluster also includes computing servers for performing computing tasks. Each computing server is equipped with a network forwarding proxy and a virtual runtime environment containing a virtual GPU, and may include: The communication unit 1501 is used to receive network packets sent by the computing server and obtain a rendering instruction stream by parsing the network packets. The network packets are generated as follows: the computing server performs address translation on the rendering instruction stream through a deployed network forwarding proxy, and then encapsulates the address-translated rendering instruction stream according to a preset network protocol to obtain the network packets, thereby converting the virtual address information contained in the rendering instruction stream into physical offsets recognizable by the rendering server. The virtual address information is used to indicate the addresses of each rendering resource associated with the rendering instruction stream within the virtual runtime environment. The rendering instruction stream is generated as follows: the computing server runs the target application in the virtual runtime environment according to the operation instructions triggered by the client for the target application, obtains the running results, and generates graphics interface call instructions corresponding to the running results for the virtual GPU, and converts the graphics interface call instructions into the rendering instruction stream. The instruction format of the rendering instruction stream is a common instruction format across multiple hardware platforms. The rendering unit 1502 is used to decode the rendering instruction stream to obtain executable drawing instructions; and execute the executable drawing instructions through the physical GPU to obtain the rendering result. The communication unit 1501 is also used to return the rendering result to the client so that the client can present the rendering result.

[0228] Optionally, the communication unit 1501 is specifically used for: The network forwarding agent receives instruction packets from the network packets via an instruction channel and resource packets from the network packets via a resource channel; wherein the instruction channel and the resource channel are pre-established communication channels between the network forwarding agent and the rendering server. The rendering unit 1502 is specifically used for: The rendering instruction stream is obtained by parsing the instruction message, and the rendering resources associated with the rendering instruction stream are obtained by parsing the resource message.

[0229] Optionally, the rendering unit 1502 is specifically used for: Based on the rendering resources, the executable drawing instructions are executed through the physical GPU to obtain the rendering result.

[0230] This application proposes a cloud rendering method for migrating cloud graphics rendering tasks based on Virtio-GPU. By running an Android virtualization environment on a computing server without a GPU and intercepting the rendering instruction stream, the method transmits the stream to a rendering server containing a physical GPU for rendering and encoding. The client only needs to decode the video stream, which enables cross-platform and low-cost graphics computing.

[0231] In addition, this application embodiment achieves physical decoupling between the Android virtualization environment and GPU rendering resources by migrating the rendering instruction stream driven by VirtIO-GPU across the network. With a three-stage pipeline of computation-rendering-encoding, it provides low-cost cloud graphics acceleration services for clients without GPUs. This not only ensures system compatibility but also enables cross-chip platform and cross-instruction set platform applications, making it suitable for graphics rendering-intensive businesses such as cloud phones and cloud games.

[0232] For the cloud rendering method provided in this application embodiment, performance analysis was performed using the glmark2 benchmark. Please refer to Table 1, which is a performance comparison table provided in this application embodiment. Table 1 shows the performance comparison between different rendering methods under 17 different load categories. The performance data in Table 1 is expressed in the format of frames per second (FPS) / frame time, where higher FPS and lower frame time are better performance. For example, the performance data of this application solution for the build load is 2152 / 0.465, that is, the FPS is 2152 and the frame time is 0.465. As can be seen from Table 1, the cloud rendering method provided in this application embodiment basically achieves about 46% of the performance of using local hardware rendering (AMD in Table 1). Due to unavoidable network overhead, its performance is slightly weaker than local hardware direct rendering, but it is more flexible than the local hardware rendering method. The Low Level Virtual Machine (LLVM) in Table 1 is a pure software simulation rendering, which is rendered on the absence of GPU hardware, and its performance is poor.

[0233] Table 1

[0234] (Performance Comparison Table)

[0235] See Figure 16 This is a schematic diagram illustrating the rendering effect provided in an embodiment of this application. The cloud rendering method provided in this embodiment renders a 3D image, and the resulting rendering result is as follows: Figure 16 As shown in the figure, the rendering backend can be seen to perform well in vertex transformation, lighting calculation and geometry output. It can be seen that the horse's surface is smooth and delicate, without obvious polygonal edges, and has high geometric accuracy. The rendering result can realistically reproduce the muscle and skin undulations of the creature.

[0236] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0237] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0238] Having introduced the cloud rendering method, apparatus, and system according to exemplary embodiments of this application, we will now introduce an electronic device according to another exemplary embodiment of this application.

[0239] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0240] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. In one embodiment, the electronic device may be a server, such as... Figure 1 The computing server 120 and rendering server 130 are shown. In this embodiment, the structure of the electronic device can be as follows: Figure 17 As shown, it includes a memory 1701, a communication module 1703, and one or more processors 1702.

[0241] The memory 1701 is used to store computer programs executed by the processor 1702. The memory 1701 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0242] Memory 1701 may be volatile memory, such as random-access memory (RAM); memory 1701 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1701 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1701 may be a combination of the above-described memories.

[0243] Processor 1702 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 1702 is used to implement the cloud rendering method described above when calling computer programs stored in memory 1701.

[0244] The communication module 1703 is used to communicate with terminal devices and other servers.

[0245] This application embodiment does not limit the specific connection medium between the memory 1701, communication module 1703, and processor 1702. This application embodiment... Figure 17 The memory 1701 and the processor 1702 are connected via a bus 1704, and the bus 1704 is in Figure 17 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1704 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 17 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0246] The memory 1701 stores a computer storage medium, which in turn stores computer-executable instructions for implementing the cloud rendering method of this application embodiment. The processor 1702 executes the aforementioned cloud rendering method.

[0247] In some possible implementations, various aspects of the cloud rendering method provided in this application can also be implemented in the form of a program product, which includes a computer program that, when the program product is run on an electronic device, causes the electronic device to perform the steps in the cloud rendering method according to the various exemplary embodiments of this application described above.

[0248] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0249] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0250] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0251] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0252] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer program can execute entirely on the user's electronic device, partially on the user's electronic device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0253] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0254] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0255] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing a computer-usable computer program.

[0256] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a mechanism for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0257] These computer program commands may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the commands stored in the computer-readable storage medium produce an article of manufacture including command means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0258] These computer program commands can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing the commands executed on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0259] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0260] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A cloud rendering method, characterized in that, A computing server is used in a cloud server cluster, wherein the computing server is deployed with a network forwarding proxy and a virtual runtime environment containing a virtual graphics processing unit (GPU); the cloud server cluster also includes a rendering server for performing rendering tasks, wherein the rendering server is deployed with a physical GPU; the method includes: Receive operation instructions triggered by the client for the target application, run the target application in the virtual runtime environment according to the operation instructions, and obtain the running results; For the virtual GPU, generate graphics interface call instructions corresponding to the running results; The graphics interface call instructions are converted into a rendering instruction stream; wherein the instruction format of the rendering instruction stream is a common instruction format across multiple hardware platforms. The network forwarding proxy performs address translation on the rendering instruction stream to convert the virtual address information contained in the rendering instruction stream into a physical offset that the rendering server can recognize; the virtual address information is used to indicate the address of each rendering resource associated with the rendering instruction stream within the virtual runtime environment. The rendered instruction stream after address translation is encapsulated into a network packet according to a preset network protocol and sent to the rendering server, so that the rendering server executes the rendered instruction stream through the physical GPU and returns the rendering result to the client for presentation.

2. The method as described in claim 1, characterized in that, The address translation of the rendering instruction stream includes: Call the address mapping function to convert the virtual address information contained in the rendering instruction stream into an address pointer accessible by the network forwarding agent; The rendering instruction stream is read through the address pointer, and the virtual resource identifier and virtual offset of each rendering resource associated with the rendering instruction stream are extracted; the virtual resource identifier is the resource identifier in the virtual runtime environment, and the virtual offset represents the offset of the corresponding rendering resource based on the resource start address in the virtual runtime environment; Based on the virtual resource identifier and virtual offset of each rendering resource, the global resource identifier and physical offset of each rendering resource are determined; the physical offset represents the offset of the corresponding rendering resource from the physical storage start address in the rendering server; the physical storage start address is located based on the global resource identifier of the corresponding rendering resource. Based on the global resource identifier and physical offset of each rendering resource, the rendering instruction stream after address translation is obtained.

3. The method as described in claim 1 or 2, characterized in that, The network forwarding agent is pre-bound to the virtual instruction queue and virtual resource queue created by the virtual GPU; Before sending the network packet to the rendering server, the method further includes: Write the rendering instruction stream into the virtual instruction queue, write each rendering resource associated with the rendering instruction stream into the virtual resource queue, and trigger a notification message for the network forwarding proxy; Sending the network packet to the rendering server includes: After receiving the notification message, the network forwarding agent reads the rendering instruction stream and each rendering resource, and encapsulates the rendering instruction stream and each rendering resource into a network packet according to a preset network protocol; The network message is sent to the rendering server.

4. The method as described in claim 3, characterized in that, The network packet includes instruction packets and resource packets; the process of encapsulating the rendering instruction stream and each rendering resource into a network packet according to a preset network protocol includes: The rendering instruction stream is encapsulated into instruction packets according to a preset network protocol, and each rendering resource is encapsulated into a resource packet according to a preset network protocol. Sending the network message to the rendering server includes: The instruction message is sent to the rendering server via the instruction channel; The resource message is sent to the rendering server through the resource channel; The instruction channel and the resource channel are pre-established communication channels between the network forwarding agent and the rendering server.

5. The method as described in claim 1 or 2, characterized in that, The cloud server cluster includes multiple rendering servers, and the network packets are sent to the rendering servers, including: Based on the latest resource usage status of each of the multiple rendering servers, a rendering server that meets the preset resource requirements is selected from the multiple rendering servers. The network message is sent to the selected rendering server.

6. The method as described in claim 1 or 2, characterized in that, Before receiving the operation command triggered by the client for the target application, the method further includes: Based on the graphics attributes of each application in the virtual runtime environment, including the target application, determine the set of virtual GPU functions that need to be deployed; Based on the aforementioned feature set, a virtual GPU is registered in the kernel of the computing server, and the virtual GPU is initialized. The initialized virtual GPU is mounted into the virtual runtime environment.

7. A cloud rendering method, characterized in that, A rendering server is used in a cloud server cluster, wherein the rendering server is deployed with a physical GPU, and the cloud server cluster also includes a computing server for performing computing tasks, wherein the computing server is deployed with a network forwarding proxy and a virtual runtime environment containing a virtual GPU, the method comprising: The system receives network packets sent by the computing server and obtains a rendering instruction stream by parsing the network packets. The network packets are generated as follows: the computing server performs address translation on the rendering instruction stream through a deployed network forwarding proxy, and then encapsulates the address-translated rendering instruction stream according to a preset network protocol to obtain the network packets, thereby converting the virtual address information contained in the rendering instruction stream into physical offsets recognizable by the rendering server. The virtual address information indicates the addresses of each rendering resource associated with the rendering instruction stream within the virtual runtime environment. The rendering instruction stream is generated as follows: the computing server runs the target application in the virtual runtime environment according to the operation instructions triggered by the client for the target application, obtains the running results, and generates graphics interface call instructions corresponding to the running results for the virtual GPU, and converts the graphics interface call instructions to obtain the rendering instruction stream. The instruction format of the rendering instruction stream is a common instruction format across multiple hardware platforms. The rendering instruction stream is decoded to obtain executable drawing instructions; The executable drawing instructions are executed by the physical GPU to obtain the rendering result; The rendering result is returned to the client so that the client can display the rendering result.

8. The method as described in claim 7, characterized in that, The step of receiving network packets sent by the computing server and obtaining the rendering instruction stream by parsing the network packets includes: The network forwarding agent receives instruction packets from the network packets via an instruction channel and resource packets from the network packets via a resource channel; wherein the instruction channel and the resource channel are pre-established communication channels between the network forwarding agent and the rendering server. The rendering instruction stream is obtained by parsing the instruction message, and the rendering resources associated with the rendering instruction stream are obtained by parsing the resource message.

9. The method as described in claim 8, characterized in that, The step of executing the executable drawing instructions through the physical GPU to obtain the rendering result includes: Based on the rendering resources, the executable drawing instructions are executed through the physical GPU to obtain the rendering result.

10. A cloud rendering system, characterized in that, It includes a client, a computing server, and a rendering server. The computing server is equipped with a network forwarding proxy and a virtual runtime environment containing a virtual GPU. The rendering server is equipped with a physical GPU. The client is used to send operation instructions triggered by the target application to the computing server; The computing server is configured to receive the operation instructions, run the target application according to the operation instructions in the virtual runtime environment, obtain the running results, generate graphics interface call instructions corresponding to the running results for the virtual GPU, and convert the graphics interface call instructions into a rendering instruction stream; the instruction format of the rendering instruction stream is a common instruction format across multiple hardware platforms. The rendering instruction stream is address translated through the network forwarding proxy to convert the virtual address information contained in the rendering instruction stream into a physical offset that the rendering server can recognize. The virtual address information is used to indicate the address of each rendering resource associated with the rendering instruction stream within the virtual runtime environment; the rendering instruction stream after address conversion is encapsulated into a network packet according to a preset network protocol, and the network packet is sent to the rendering server; The rendering server is configured to receive network packets sent by the computing server and obtain a rendering instruction stream by parsing the network packets; decode the rendering instruction stream to obtain executable drawing instructions; execute the executable drawing instructions through the physical GPU to obtain a rendering result; and return the rendering result to the client. The client is also used to present the rendering result.

11. The system as claimed in claim 10, characterized in that, The system includes multiple rendering servers and a resource scheduler, which is used for: Based on the latest resource usage status of each of the multiple rendering servers, a rendering server that meets the preset resource requirements is selected from the multiple rendering servers, so that the computing server sends the network packet to the selected rendering server.

12. A cloud rendering device, characterized in that, A computing server used in a cloud server cluster, wherein the computing server is deployed with a network forwarding proxy and a virtual runtime environment containing a virtual GPU, and the cloud server cluster also includes a rendering server for performing rendering tasks, wherein the rendering server is deployed with a physical GPU; the device includes: A computing unit is configured to receive operation instructions triggered by a client for a target application, run the target application according to the operation instructions in the virtual runtime environment, and obtain the running results; generate graphics interface call instructions corresponding to the running results for the virtual GPU; and convert the graphics interface call instructions into a rendering instruction stream; wherein the instruction format of the rendering instruction stream is a common instruction format across multiple hardware platforms. The communication unit is configured to perform address translation on the rendering instruction stream via the network forwarding proxy, so as to convert the virtual address information contained in the rendering instruction stream into a physical offset that can be recognized by the rendering server; the virtual address information is used to indicate the address of each rendering resource associated with the rendering instruction stream within the virtual runtime environment; the address-translated rendering instruction stream is encapsulated into a network packet according to a preset network protocol, and the network packet is sent to the rendering server, so that the rendering server executes the rendering instruction stream through the physical GPU and returns the rendering result to the client for presentation.

13. A cloud rendering device, characterized in that, A rendering server used in a cloud server cluster, the rendering server being deployed with a physical GPU, the cloud server cluster also including a computing server for performing computing tasks, the computing server being deployed with a virtual runtime environment containing a virtual GPU, the device comprising: A communication unit is configured to receive network packets sent by the computing server and obtain a rendering instruction stream by parsing the network packets. The network packets are generated as follows: the computing server performs address translation on the rendering instruction stream through a deployed network forwarding proxy, and then encapsulates the address-translated rendering instruction stream according to a preset network protocol to obtain the network packets, thereby converting the virtual address information contained in the rendering instruction stream into physical offsets recognizable by the rendering server. The virtual address information indicates the addresses of each rendering resource associated with the rendering instruction stream within the virtual runtime environment. The rendering instruction stream is generated as follows: the computing server runs the target application in the virtual runtime environment according to the operation instructions triggered by the client for the target application, obtains the running results, and generates graphics interface call instructions corresponding to the running results for the virtual GPU, and converts the graphics interface call instructions into the rendering instruction stream. The instruction format of the rendering instruction stream is a common instruction format across multiple hardware platforms. The rendering unit is used to decode the rendering instruction stream to obtain executable drawing instructions; and to execute the executable drawing instructions through the physical GPU to obtain the rendering result. The communication unit is also used to return the rendering result to the client so that the client can present the rendering result.

14. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any of the methods of claims 1 to 6, or to perform the steps of any of the methods of claims 7 to 9.

15. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods of claims 1 to 6, or to perform the steps of any of the methods of claims 7 to 9.

16. A computer program product, characterized in that, The method includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the methods of claims 1 to 6, or to perform the steps of any of the methods of claims 7 to 9.

Images