Task processing method and system, electronic device, and storage medium

Abstract

Embodiments of the present application provide a task processing method and system, an electronic device, and a storage medium. The method comprises: in response to a request for processing a target task, a client virtual machine generating a target command; the client virtual machine sending the target command to an adapter configured in a virtual graphics processing unit (GPU) device of a host machine; by means of the adapter, parsing the target command to obtain command requirement information corresponding to the target command, and on the basis of the command requirement information, determining task processing resource availability information of the target task; and in response to the determination of the task processing resource availability information, the adapter allowing a corresponding task processing device to process the target task on the basis of the task processing resource availability information, so as to obtain a processing result.

Description

Task processing method, system, electronic device and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on September 21, 2023, with application number 202311229694.1 and application name “Task processing method, system, electronic device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of computer virtualization technology, and in particular to a task processing method, system, electronic device and storage medium. Background Art

[0004] A virtual graphics processing unit (GPU) is a GPU device implemented using virtualization technology, which provides GPU-related functions to virtual machines, such as 3D rendering, audio and video encoding and decoding, and parallel computing acceleration.

[0005] GPU virtualization technology is generally provided directly by GPU manufacturers. This technology uses the method of logical segmentation of physical resources to divide the physical GPU device into several virtual GPU devices, and is supported by its own closed-source GPU driver software.

[0006] Summary of the Invention

[0007] Each exemplary embodiment of the present application provides a task processing method, system, electronic device, and storage medium.

[0008] [Corrected on 24.02.2025 according to Rule 91] According to the first aspect of the present application, a task processing method is provided, comprising: a client virtual machine generates a target command in response to a processing request of a target task; the client virtual machine sends the target command to an adapter provided in a virtual graphics processor (GPU) device of a host machine; command requirement information corresponding to the target command is parsed from the target command by the adapter, and task processing resource availability information of the target task is confirmed based on the command requirement information; and in response to the confirmation of the task processing resource availability information, the adapter allows the corresponding task processing device to process the target task based on the task processing resource availability information to obtain a processing result.

[0009] The exemplary embodiment of the present application realizes abstract management of physical GPU resources by sending target commands through the client virtual machine and the adapter set in the virtual GPU device to parse the target commands, and supports free on-demand allocation and dynamic scaling of physical GPU resources. It can flexibly allocate resources, improve the timeliness of processing results, and improve the resource utilization rate of the physical GPU device.

[0010] In one embodiment, the adapter allows the corresponding task processing device to process the target task based on the task processing resource availability information, including: the adapter instructs the corresponding task processing device to allocate task processing resources for the target task based on the task processing resource availability information.

[0011] In one embodiment, before the adapter allows the corresponding task processing device to process the target task based on the task processing resource availability information, the method further includes: the adapter obtaining a processing load status of the corresponding task processing device; and in response to the processing load status being greater than or equal to a first threshold, the adapter suspending allowing the corresponding task processing device to process the target task based on the task processing resource availability information.

[0012] In one embodiment, in response to the processing load state being greater than or equal to a first threshold, the adapter suspends the corresponding task processing device from processing the target task based on the task processing resource availability information, including: in response to the number of vertices of resources to be rendered by the corresponding task processing device within a preset time exceeding a second threshold, the adapter suspends the corresponding task processing device from processing the target task based on the task processing resource availability information.

[0013] In one embodiment, the target command includes at least one of a rendering command, a video acceleration command, or a parallel computing command.

[0014] In one embodiment, the client virtual machine generates the target command in response to the processing request of the target task, including: generating the target command in response to the processing request of the target task by a target processor set in a user-mode driver of the client virtual machine, wherein the target processor is a processor that matches the target task among one or more processors in the client virtual machine.

[0015] In one embodiment, parsing the command requirement information from the target command through the adapter includes: allocating the target command to the target scheduling unit of the resource scheduling module through the resource scheduling module of the adapter, wherein the target scheduling unit is a scheduling unit corresponding to the target task among one or more scheduling units of the resource scheduling module; and parsing the command requirement information from the target command through the target scheduling unit.

[0016] In one embodiment, the target processor generates the target command in response to the processing request of the target task, and the client virtual machine sends the target command to the adapter set in the GPU device of the host machine, including: if the target application is the rendering application, the rendering processor generates the rendering command in response to the processing request of the target task initiated by the rendering application, and the client virtual machine sends the rendering command to the adapter; if the target application is the video application, the video processor generates the video acceleration command in response to the processing request of the target task initiated by the video application, and the client virtual machine sends the video acceleration command to the adapter; and if the target application is the parallel computing application, the parallel computing processor generates the parallel computing command in response to the processing request of the target task initiated by the parallel computing application, and the client virtual machine sends the parallel computing command to the adapter.

[0017] In one embodiment, parsing the command requirement information from the target command by the target scheduling unit includes: if the target command is the rendering command, parsing the rendering requirement information of the rendering task from the rendering command by the rendering scheduling unit; if the target command is the video acceleration command, parsing the video requirement information of the video task from the video acceleration command by the video scheduling unit; and if the command is the parallel computing command, parsing the computing requirement information of the parallel computing task from the parallel computing command by the parallel computing scheduling unit.

[0018] In one embodiment, the method further includes: converting, by the rendering processor, a target task initiated by the rendering application in the first system environment into a rendering task supported in the second system environment; wherein, the first system environment is different from the second system environment, and the corresponding task processing device supports the second system environment.

[0019] In one embodiment, the client virtual machine sends the target command to the adapter set in the GPU device of the host machine, including: the client virtual machine sends the target command to the adapter set in the GPU device of the host machine through a kernel state driver set in the client virtual machine.

[0020] In one embodiment, the kernel-mode driver includes at least a command cache and a data cache, and the method further includes: obtaining the target command from the target processor through the command cache; scheduling a pending command queue through the command cache, wherein the target command is cached in the pending command queue; and caching data of the target task through the data cache.

[0021] In one embodiment, the adapter further includes: a command cache module and a data cache module, and the method further includes: caching the target command obtained from the command cache area through the command cache module; and caching the data of the target task obtained from the data cache area and the processing result of the target task through the data cache module.

[0022] In one embodiment, the task processing resource availability information is confirmed based on physical GPU resource availability information and CPU resource availability information.

[0023] In one embodiment, confirming the task processing resource availability information of the target task based on the command requirement information includes: if the GPU resource availability information is greater than or equal to the command requirement information, confirming the GPU resource availability information as the task processing resource availability information; if the GPU resource availability information is less than the command requirement information and is higher than or equal to a third threshold, confirming the GPU resource availability information and the CPU resource availability information as the task processing resource availability information, wherein the third threshold is less than the command requirement information; and if the GPU resource availability information is less than the third threshold, confirming the CPU resource availability information as the task processing resource availability information.

[0024] In one embodiment, the corresponding rendering device includes at least one of a physical GPU device and a CPU device.

[0025] According to a second aspect of the present application, a task processing method is provided, comprising:

[0026] A target processor set in a user-mode driver of a client virtual machine generates a target command in response to a processing request of a target task, and sends the target command to an adapter set in a virtual GPU device through a kernel-mode driver set on the client virtual machine; wherein the target processor is a processor among multiple processors that matches the target application; the target task is a GPU task initiated by the target application, the target command is a command corresponding to the target task, and the target command includes physical GPU resource requirement information and / or CPU resource requirement information required to process the target task; the target command is allocated to a target scheduling unit through a resource scheduling module; wherein the target scheduling unit is a scheduling unit among multiple scheduling units that matches the target application; the target scheduling unit parses command requirement information from the target command, and allocates physical GPU resource availability information and / or CPU resource availability information according to the command requirement information; and when the physical GPU resource availability information and / or the CPU resource availability information are allocated, the physical GPU device and / or the CPU device processes the target task to obtain a processing feedback result.

[0027] In one embodiment, the target application includes: at least one of a rendering application, a video application, and a parallel computing application; the target processor includes: at least one of a rendering processor, a video processor, and a parallel computing processor; the target scheduling unit includes: at least one of a rendering scheduling unit, a video scheduling unit, and a parallel computing scheduling unit; or the target command includes: at least one of a rendering command, a video acceleration command, and a parallel computing command.

[0028] In one embodiment, the generating of the target command by the target processor in response to the processing request of the target task, and sending the target command to the adapter through the kernel state driver, includes at least one of the following: generating the rendering command by the rendering processor in response to the processing request of the rendering task initiated by the rendering application, and sending the rendering command to the adapter through the kernel state driver; generating the video acceleration command by the video processor in response to the processing request of the video task initiated by the video application, and sending the video acceleration command to the adapter through the kernel state driver; or generating the parallel computing command by the parallel computing processor in response to the processing request of the parallel computing task initiated by the parallel computing application, and sending the parallel computing command to the adapter through the kernel state driver.

[0029] In one embodiment, the parsing of command requirement information from the target command by the target scheduling unit includes at least one of the following: parsing rendering requirement information of a rendering task from the rendering command by the rendering scheduling unit; parsing video requirement information of a video task from the video acceleration command by the video scheduling unit; or parsing computing requirement information of a parallel computing task from the parallel computing command by the parallel computing scheduling unit.

[0030] In one embodiment, the method further includes: converting, by the rendering processor, a target task initiated by the rendering application in the first system environment into a supported rendering task in the second system environment; wherein the first system environment is different from the second system environment, and the corresponding task processing device supports the second system environment.

[0031] In one embodiment, the kernel-mode driver includes a command cache and a data cache; the method further includes: caching the target commands obtained from the target processor through the command cache, and scheduling a queue of pending commands; wherein the queue of pending commands is composed of the cached target commands; and caching the data of the target task through the data cache.

[0032] In one embodiment, the adapter further comprises: a command cache module and a data cache module;

[0033] The method further comprises:

[0034] caching the target command obtained from the command buffer area by the command buffer module; and

[0035] The data cache module caches the target task data obtained from the data cache area and the processing feedback result of the target task.

[0036] In one embodiment, the virtual GPU device is further configured to perform any of the following:

[0037] Allocate physical GPU resources; wherein the physical GPU device corresponding to the virtual GPU device has a memory vram for managing the allocation of physical GPU resources;

[0038] After command parsing, rendering and / or computing resources are allocated according to the resource requirements of the target task corresponding to the target command, and the resources include physical GPU resources and virtual GPU resources;

[0039] Processing a command list in the command cache module, wherein the command list is a command stream in the command cache module and is used to cache rendering commands, video acceleration commands, and parallel computing commands;

[0040] Processing frame synchronization information, wherein the frame synchronization information is used to implement video acceleration processing and / or 3D rendering processing;

[0041] Managing the state of target processing operations including rendering, parallel computing, and video processing; or

[0042] Managing the physical GPU resources.

[0043] According to a third aspect of the present application, a task processing device is provided, comprising: a client virtual machine, configured to generate a target command in response to a processing request of a target task, and configured to send the target command to an adapter provided in a virtual graphics processor (GPU) device of a host machine; and the adapter, configured to parse command requirement information from the target command, and confirm task processing resource availability information of the target task based on the command requirement information; and configured to allow a corresponding task processing device to process the target task based on the task processing resource availability information in response to confirmation of the task processing resource availability information, so as to obtain a processing result.

[0044] According to a fourth aspect of the present application, a task processing system is provided; the task processing system includes: a virtual GPU device located on a host machine, and a target application and a virtual GPU driver deployed on a client virtual machine, the virtual GPU driver virtual program including a user-mode driver and a kernel-mode driver; wherein the user-mode driver is provided with multiple processors; the virtual GPU device is provided with an adapter, the adapter including a resource scheduling module, and the resource scheduling module including multiple scheduling units;

[0045] a target processor, configured to generate a target command in response to a processing request of a target task, and send the target command to the adapter through the kernel-mode driver; wherein the target processor is a processor among the multiple processors that matches the target application; the target task is a GPU task initiated by the target application, the target command is a command corresponding to the target task, and the target command includes physical GPU resource requirement information and / or CPU resource requirement information required to process the target task;

[0046] The resource scheduling module is configured to allocate the target command to a target scheduling unit; wherein the target scheduling unit is a scheduling unit that matches the target application among the multiple scheduling units;

[0047] The target scheduling unit is configured to parse command requirement information from the target command and allocate physical GPU resource availability information and / or CPU resource availability information according to the command requirement information;

[0048] When the physical GPU resource availability information is allocated, the physical GPU device processes the target task and obtains a processing feedback result.

[0049] According to a fifth aspect of the present application, there is provided an electronic device, comprising: at least one processor and a memory;

[0050] The memory stores computer-executable instructions;

[0051] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the task processing method described in the first aspect above.

[0052] According to a sixth aspect of the present application, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the task processing method described in the first aspect above.

[0053] According to a seventh aspect of the present application, a computer program product is provided, comprising a computer program, which implements the task processing method described in the first aspect when executed by a processor.

[0054] The present application provides a task processing method, comprising: generating a target command in response to a processing request of a target task by a target processor, and sending the target command to an adapter through a kernel-mode driver; wherein the target processor is a processor that matches a target application among multiple processors; the target task is a GPU task initiated by a target application, and the target command is a command corresponding to the target task; allocating the target command to a target scheduling unit through a resource scheduling module; wherein the target scheduling unit is a scheduling unit that matches the target application among multiple scheduling units; parsing command requirement information from the target command by the target scheduling unit, and allocating physical GPU resource availability information and / or CPU resource availability information according to the command requirement information; when the physical GPU resource availability information and / or CPU resource availability information is allocated, processing the target task through a physical GPU device and / or CPU device to obtain a processing feedback result.

[0055] This application can achieve abstract management of physical GPU resources through the target processor set in the user-mode driver of the client virtual machine and the adapter set in the virtual GPU device, and support the free allocation and dynamic scaling of physical GPU resources. It can flexibly allocate resources, improve the timeliness of processing feedback results, and improve the resource utilization rate of the physical GPU device.

[0056] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0058] FIG1 is a schematic diagram of a rendering process in an operating system in the related art.

[0059] FIG2 is a flowchart of Windows graphics operation in related art.

[0060] FIG3 is a flowchart of Linux graphics operation in related art.

[0061] FIG4 is a first schematic diagram of a first virtual GPU implementation method in the related art.

[0062] FIG5 is a second schematic diagram of the first virtual GPU implementation method in the related art.

[0063] FIG6 is a schematic diagram of a second virtual GPU implementation method in the related art.

[0064] FIG7 is a schematic diagram of a third virtual GPU implementation method in the related art.

[0065] FIG8 is a schematic diagram of the structure of a task processing system provided in one embodiment of the present application.

[0066] FIG9 is a schematic diagram of a processing flow of a task processing system provided in an embodiment of the present application.

[0067] FIG10 is a schematic diagram of a processing flow of a task processing system provided in another embodiment of the present application.

[0068] FIG11 is a schematic diagram of a flow chart of various calling interfaces, a scheduling unit, and a virtual GPU device provided in an embodiment of the present application.

[0069] FIG12 is a schematic diagram of a scheduling process provided in an embodiment of the present application.

[0070] FIG13 is a schematic diagram of the structure of a virtual GPU architecture provided in an embodiment of the present application.

[0071] FIG14 is a flowchart of a task processing method provided in an embodiment of the present application.

[0072] FIG15 is a flowchart of a task processing method provided in an embodiment of the present application.

[0073] FIG16 is a schematic diagram of a task processing device provided in an embodiment of the present application.

[0074] FIG17 is a schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0075] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of the technical solutions of this application, but not all of them. Based on the embodiments described in this application document, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the technical solutions of this application.

[0076] In addition, the terms "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, systems, products, or devices. "Multiple" in this application can mean at least two, for example, two, three, or more, and the embodiments of this application are not limited thereto.

[0077] In the description of this application, "multiple" is understood to mean "at least two." "And / or" describes the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. A and B are connected, which can mean: A and B are directly connected, and A and B are connected through C. In addition, in the description of this application, words such as "first" and "second" are used only for the purpose of distinguishing descriptions and should not be understood as indicating or implying relative importance or order.

[0078] Each exemplary embodiment of the present application provides a task processing method, system, electronic device, and storage medium.

[0079] To facilitate understanding, the application scenarios of each exemplary embodiment of the present application are first introduced.

[0080] With the help of the hypervisor in virtualization technology, users can create multiple simulated environments or dedicated resources based on a single physical hardware system. The hypervisor can be directly connected to the physical hardware system, thereby dividing a physical hardware system into different, separate and secure environments, namely virtual machines (VMs). The hypervisor can separate computer resources from the hardware and allocate resources appropriately, which is a very important function for virtual machines. Generally, the physical hardware equipped with a hypervisor is called a host, and the virtual machines that use its resources are called client virtual machines, virtual clients, etc. These client virtual machines regard computing resources (such as CPU, memory, and storage) as a set of resources that can be reallocated. Operators can control virtual instances of CPU, memory, storage, and other resources so that client virtual machines can receive the required resources when needed.

[0081] A virtual graphics processor (vGPU) is a physical GPU device implemented through virtualization technology, which provides GPU-related functions to virtual machines, such as 3D rendering, audio and video encoding and decoding, and parallel computing acceleration. This GPU virtualization technology is generally provided directly by GPU manufacturers. This technology uses a method of logical segmentation of physical resources to divide the physical GPU device into several virtual GPU devices, and its own closed-source GPU driver software provides functional support. This type of technology cannot adapt well to the needs of elastic resource allocation in cloud computing. It is divided into a fixed number and cannot be changed dynamically. In addition, virtual machine migration can only be performed between computing nodes using the same model of GPU, which seriously limits the reasonable distribution of computing resources. Therefore, this technology cannot perform flexible physical GPU resource allocation, and the utilization rate of physical GPU resources is low, which leads to poor timeliness of task processing.

[0082] For example, in the cloud desktop field, all virtual desktop infrastructure (VDI) solutions that support GPUs use the aforementioned GPU virtualization solution based on hardware segmentation. This solution is relatively expensive and limits the way physical GPU resources are allocated. Cloud desktop vendors are unable to customize more reasonable management strategies based on this solution.

[0083] In the cloud computing field, infrastructure platforms based on independent virtualization solutions such as the Quick Emulator (QEMU) and open-source Kernel-based Virtual Machine (KVM) have gained widespread adoption, enabling flexible management of computing resources across ultra-large computer clusters. However, due to the closed nature of GPU technology, virtual GPUs lack a unified technical approach and are not widely adopted.

[0084] Some GPU vendors' virtual GPUs are tied to physical GPUs, relying solely on hardware logic. This makes them incompatible with the flexible and elastic nature of cloud computing. Cloud environments require 3D rendering, such as in film and animation production, as well as audio and video encoding and decoding, such as in live video streaming services, and parallel computing, such as in AI services. The need for virtual GPUs is particularly acute in the cloud desktop space, where graphical desktop operations rely on GPU support for smooth operation. Conventional vGPUs are expensive and difficult to deploy on a large scale in the cloud.

[0085] Figure 1 is a schematic diagram of the rendering process in an operating system in the related art. As shown in Figure 1, applications using graphics rendering need to call relevant rendering call interfaces (Application Programming Interfaces, APIs), such as OpenGL applications, Vulkan applications, and Direct3D applications. These APIs are in turn supported by dedicated GPU drivers. In Windows and Linux systems, the GPU driver framework mainly consists of a user-mode driver (UMD) and a kernel-mode driver (KMD). The UMD is responsible for rendering resource management and computing resource allocation, while the KMD is responsible for directly controlling the GPU device.

[0086] Figure 2 is a flowchart of Windows graphics operations in related technology. In Windows, the GPU driver framework is called wddm, UMD mainly implements the requirements of Direct3D applications, KMD is the port miniport driver requirement, and the graphics subsystem dxgk in Windows completes tasks such as rendering task scheduling and inter-process communication.

[0087] Figure 3 is a flowchart of Linux graphics operations in related art. As shown in Figure 3, in Linux, UDM mainly implements OpenGL support, which is supported by mesa. KDM is called kernel drm in Linux, and the display part, mesa, and drm are collectively called dri.

[0088] As shown in FIG. 4 to FIG. 7 , the related technical solutions provide three methods for implementing GPU virtualization technology.

[0089] Figure 4 is a first schematic diagram of a first virtual GPU implementation method in the related art. This first virtual GPU implementation method is based on hardware-segmented GPU virtualization technology. Taking the AMD vGPU solution as an example, this solution provides virtual GPUs by directly segmenting resources from the physical GPU device. Specifically, the host sets parameters for the physical GPU device's support for the virtual GPU device; AMD provides a vGPU driver running within the virtual machine; and the virtual GPU driver directly controls and manages the virtual GPU device provided by the physical GPU.

[0090] FIG5 is a second schematic diagram of a first virtual GPU implementation method of the related art, which is a hardware-based virtual GPU implementation method applied to the GPU main management layer loaded in the host kernel. The method includes: dividing the shared resources of the graphics subsystem into multiple resource regions; during the host kernel startup phase, allocating a resource region to the core GPU driver layer loaded in the host kernel; during each virtual machine kernel startup phase, allocating a resource region to the core GPU driver layer loaded in each virtual machine kernel; obtaining and sequentially forwarding service requests sent by the sender, which includes the host and / or virtual machine, and receiving and returning service responses from the GPU. In this technical solution, the service request from the sender (virtual machine and / or host) only needs to be forwarded through the GPU main management layer loaded in the kernel, without having to be copied and called through multiple intermediate layers.

[0091] The first virtual GPU implementation method in related technologies has two problems. One is that it is not flexible enough and cannot dynamically change the allocation of physical GPU resources, that is, it cannot be flexibly deployed. The other is that it is usually expensive.

[0092] Figure 6 is a schematic diagram of a second virtual GPU implementation method in the related art. As shown in Figure 6, this second virtual GPU implementation method is based on API forwarding technology. The key to this method is the implementation of an OpenGL wrapper, which connects the Linux in the virtual machine with OpenGL on the host Linux system via remote procedure calls (RPCs), with OpenGL on the host actually executing rendering commands. In the Linux virtual machine, the application calls the OpenGL API, which is then implemented by the OpenGL runtime. The application then enters the OpenGL wrapper, which then calls the host OpenGL API via remote procedure calls (RPCs). After entering the OpenGL runtime, the host parses OpenGL parameters, processes relevant data, and submits it to the Mesa. After entering the user-mode driver (Mesa), the user-mode driver (Mesa) actually handles all OpenGL rendering requests, generating rendering commands and data. The kernel-mode driver (DRM) reads and processes the user-mode driver data and submits rendering commands to the GPU.

[0093] Figure 7 is a schematic diagram of a third virtual GPU implementation method in the related art. As shown in Figure 7, the third virtual GPU implementation method in the related art is the virtio-gpu solution. This method is implemented based on the Linux system and works on Linux. The key to its implementation lies in: user-mode drivers and vGPU rendering functions. Specifically, this solution implements the vGPU virtual device operating logic on the host, independently implementing rendering-related functions as a renderer; modifies the user-mode and kernel-mode drivers in the Linux system running in the virtual machine to add support for vGPU; and implements the actual OpenGL functional logic in the renderer.

[0094] In summary, the related technologies have the following problems:

[0095] (1) The relevant technology cannot fully adapt to the GPU functional requirements of virtual machines.

[0096] (2) The application interface usually directly instructs the physical GPU device to execute GPU functions. Therefore, the call of GPU functions must be directly bound to the physical GPU device of a specific manufacturer. It is impossible to adaptively use physical GPU devices of different manufacturers, and thus cannot support the adaptation of physical GPU devices of multiple manufacturers, which ultimately leads to the inability to flexibly implement the allocation of physical GPU resources.

[0097] (3) The relevant technology can only provide support for 3D rendering applications on certain operating systems, but not for certain applications. For example, it cannot provide support for Direct3D applications.

[0098] (4) Some virtual GPU implementations, such as the first virtual GPU implementation mentioned above, are bound to physical GPU devices and provided by GPU manufacturers. However, technically, they cannot support flexible resource allocation customization requirements and are difficult to adapt to the changing cloud computing environment.

[0099] (5) Some virtual GPU implementation solutions, such as remotegl and wiregl, achieve some of the functions provided by the physical GPU by intrusively modifying driver functions or hooking various applications through software of the Windows message processing mechanism. Such solutions cannot fully adapt to the functional requirements of the virtual machine guest system.

[0100] (6) Some virtual GPU implementations, such as the third virtual GPU implementation virtio-gpu, can only be supported by the 3D rendering API on certain operating systems.

[0101] (7) Rendering standards are often different in different operating systems. For example, Direct3D applications can run on Windows, but cannot run normally on Linux. Currently, there is no relatively complete rendering conversion solution.

[0102] Each exemplary embodiment of the present application provides a task processing method, apparatus, system, electronic device, computer-readable storage medium, and computer program product applied to the field of computer virtualization technology for improving resource utilization of a physical GPU device.

[0103] It should be noted that the exemplary embodiments of this application belong to the virtual GPU technology in the field of computer virtualization technology. By designing the GPU driver in the operating system and cooperating with the virtual GPU device, the full virtualization of the physical GPU device is achieved, and the functions of all physical GPU devices are supported in the client virtual machine, including 3D rendering, video codec acceleration, parallel computing acceleration, etc.

[0104] The technical solutions for virtual GPU drivers and virtual GPU device implementations provided by the exemplary embodiments of this application are applicable to a wide range of scenarios, including but not limited to: cloud computing clusters, cloud desktops, and virtual machines within personal computers (PCs). They can be deployed in scenarios such as higher education, general education, and finance, as well as various scenarios that require GPU-supported computing acceleration functions such as 3D rendering and video acceleration.

[0105] The following exemplary embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The exemplary embodiments described below may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0106] FIG8 is a schematic diagram of the structure of a task processing system provided in an embodiment of the present application. As shown in FIG8 , the task processing system includes a client virtual machine and a host machine, wherein the client virtual machine utilizes the physical computing resources on the host machine. The task processing system also includes: a virtual GPU device 2 located on the host machine, a target application 3 deployed on the client virtual machine, and a virtual GPU driver. The virtual GPU driver includes a user-mode driver 4 and a kernel-mode driver 5.

[0107] One or more processors may be provided in the user mode driver 4. The virtual GPU device 2 is provided with an adapter 21, which is used to dispatch corresponding physical computing resources to process the target based on the type of the received task command.

[0108] The adapter 21 may include a resource scheduling module 211, which includes multiple scheduling units. The resource scheduling module 211 is used to schedule relevant physical computing resources according to the task commands of the vGPU, and the multiple scheduling units are used to schedule corresponding physical computing resources corresponding to different types of task commands.

[0109] Although FIG8 shows the physical GPU resources scheduled by the target scheduling unit 2111 , it is understandable that the target scheduling unit 2111 may also schedule physical CPU resources.

[0110] It should be understood that the host machine can be referred to as the host, the client virtual machine can also be referred to as a guest, and the virtual GPU device 2 can also be referred to as the vGPU device 2. The user-mode driver 4 can also be referred to as a user-level driver, and the kernel-mode driver 5 can also be referred to as a kernel-level driver.

[0111] User-mode driver 4 includes a target processor 41. Target processor 41 is configured to generate a target command in response to a processing request from a target task and send the target command to adapter 21 via kernel-mode driver 5. Target processor 41 may be one or more processors that matches target application 3. A target task is a GPU processing task initiated by target application 3. A target command is a command corresponding to the target task, and the target command includes information about available task processing resources required to process the target task (e.g., information about physical GPU resource requirements or CPU resource requirements).

[0112] The resource scheduling module 211 is used to allocate the target command to the target scheduling unit 2111; wherein the target scheduling unit 2111 is a scheduling unit that matches the target application 3 among multiple scheduling units.

[0113] The target scheduling unit 2111 is used to obtain the task processing resource availability information of the physical GPU device 1 (for example, it may include at least one of the physical GPU resource requirement information or the CPU resource requirement information), parse the command requirement information from the target command, and allocate the physical GPU resource availability information and / or the CPU resource availability information according to the command requirement information.

[0114] When the physical GPU resource availability information is allocated, the corresponding physical GPU resource of the physical GPU device 1 is called, and the physical GPU device 1 processes the target task to obtain a processing result (for example, at least one of a rendering result, a video acceleration result, or a parallel computing result).

[0115] The embodiment of the present application can achieve flexible and abstract management of physical GPU resources through the target processor set in the user-mode driver of the client virtual machine and the adapter set in the virtual GPU device, support the free allocation and dynamic scaling of physical GPU resources, and can flexibly allocate resources, thereby improving the resource utilization rate of the physical GPU device while improving the timeliness of processing feedback results.

[0116] 9 , the target application 3 may include a rendering application 31 , a video application 32 , and a parallel computing application 33 . In other embodiments, the target application 3 may also include at least one of the rendering application 31 , the video application 32 , and the parallel computing application 33 .

[0117] It should be understood that the above-mentioned rendering application 31 or rendering software, as shown in FIG10 , may include: OpenGL application, Vulkan application, Direct3D application, etc.

[0118] 9 , the target processor 41 may include a rendering processor 411 , a video processor 412 , and a parallel computing processor 413 . In other embodiments, the target processor 41 may also include at least one of the rendering processor 411 , the video processor 412 , and the parallel computing processor 413 .

[0119] It should be understood that the video processor 412 may also be referred to as a video acceleration processor 412 , and the parallel computing processor 413 may also be referred to as a parallel computing acceleration processor.

[0120] As shown in Figure 9, the target scheduling unit 2111 may include: a rendering scheduling unit 221, a video scheduling unit 222, and a parallel computing scheduling unit 223. In other embodiments, the target scheduling unit 2111 may also include at least one of the rendering scheduling unit 221, the video scheduling unit 222, or the parallel computing scheduling unit 223.

[0121] That is, the adapter 21 includes a resource scheduling module 211, and the resource scheduling module 211 includes multiple scheduling units, namely, a rendering scheduling unit 221, a video scheduling unit 222, and a parallel computing scheduling unit 223. The adapter 21 is used to schedule the computing resources of the physical GPU device 1 through corresponding data calculation according to the target command.

[0122] In this embodiment, the target command may include: a rendering command, a video acceleration instruction (including video acceleration parameters) and a parallel computing command. In other embodiments, the target command may include at least one of a rendering command, a video acceleration command or a parallel computing command.

[0123] The embodiments of the present application provide a flexible and comprehensive virtual GPU device for client virtual machines, achieving full-featured virtualization of the virtual GPU device, including various functions such as computational acceleration, video acceleration, and 3D rendering. Through the provided virtual GPU device, the embodiments enable user-mode drivers to call target applications, thereby enabling the call of physical GPU devices from different specific vendors by switching client virtual machines.

[0124] Based on the usage of physical GPU resources, this embodiment can flexibly allocate physical GPU resources so that physical GPU resources in cloud computing can be dynamically scheduled, which can not only greatly improve the utilization efficiency of physical GPU devices but also reduce the deployment cost of physical GPU devices.

[0125] The following describes the functions and corresponding modules of the virtual GPU device 2.

[0126] The target processor 41 is configured to generate a target command in response to a processing request corresponding to a target task, and send the target command to the adapter 21 via the kernel-mode driver 5. As shown in FIG10 , the target processor 41 may include a rendering processor 411, a video processor 412, and a parallel computing processor 413.

[0127] The rendering processor 411 generates a rendering command in response to a processing request for a rendering task initiated by the rendering application 31 , and sends the rendering command to the adapter 21 through the kernel-mode driver 5 .

[0128] The video processor 412 generates a video acceleration instruction in response to a video task processing request initiated by the video application 32, and sends the video acceleration instruction to the adapter through a kernel-mode driver.

[0129] The parallel computing processor generates a parallel computing command in response to a processing request for a parallel computing task initiated by a parallel computing application, and sends the parallel computing command to the adapter through a kernel-mode driver.

[0130] In other embodiments, the target processor 41 may also include at least one of a rendering processor 411 , a video processor 412 , or a parallel computing processor 412 .

[0131] It should be understood that the above-mentioned video acceleration instructions include video parameters or video parameter information, which are used to perform video acceleration tasks.

[0132] As shown in FIG10 , the rendering processor 411 can be set in the user-mode driver of the client virtual machine, and a corresponding rendering scheduling unit 211 cooperates with it in the virtual GPU device 2 to manage and process the rendering tasks initiated by the client virtual machine.

[0133] In this embodiment, the function of the rendering processor can be realized through a user-mode driver, a kernel-mode driver, and a driver loader (not shown in the figure). The driver loader is used to select the driver entry function address corresponding to the rendering processor. In response to the target task received by the user-mode driver, the user-mode driver starts the rendering manager, executes the processing flow of the target task initiated by the OpenGL application, Vulkan application, Direct3D application, etc., and manages the physical GPU resources / CPU resources through the physical GPU resource requirement information and / or CPU resource requirement information. The user-mode driver is uniformly issued the drawcall command and resource data (i.e., the above-mentioned physical GPU resource requirement information and / or CPU resource requirement information) by the rendering manager.

[0134] As shown in Figures 10 and 11, the video processor can be set in the user-mode driver of the client virtual machine, and the video processor can have the following functions. (1) Receive calls to the video processor through the calling interface. (2) The video processor receives data called from the video application. (3) Parse the data and process the video acceleration command. (4) Call the function of the rendering manager to generate the picture buffer. (5) According to the video acceleration command, call the feature processing unit to support multiple mainstream video formats. (6) Parse data units such as macroblocks to support multiple mainstream video formats. (7) Call the corresponding encoder or decoder function according to functional requirements and video type to support multiple mainstream video formats. (8) Generate the corresponding encoding / decoding command stream. (9) Write the corresponding video frame data to the frame buffer (i.e., data buffer).

[0135] In one embodiment, the video processor can also be used to start a parallel computing processor to receive data and call the parallel computing processor for fast computing. Furthermore, the parallel computing processor can be provided with a computing adapter, which is used to receive data and call the parallel computing processor for fast computing.

[0136] As shown in Figures 10 and 11, the parallel computing processor is set in the user-mode driver of the client virtual machine and can have the following functions: (1) Receive calls for parallel computing applications. (2) Generate computing kernels. (3) Call the shader in the rendering manager to generate intermediate representation language. (4) Call the rendering manager to process image generation. (5) Allocate computing resources corresponding to the required data. (6) Submit computing commands and data.

[0137] Compared to related virtual GPU implementations, the virtual GPU implementation in this embodiment utilizes a fully functional virtual GPU device. This device integrates with the Windows system's wddm driver framework (including the user-mode driver wddm umd and user-mode driver wddm umd), supports standard rendering and video codec functionality on Windows, and ensures the overall functional integrity of the client virtual machine. This integrity is reflected in its complete functionality and compatibility with multiple operating systems.

[0138] In one possible implementation, the rendering processor 411 may also be configured to convert a target task initiated by the rendering application 31 in a first system environment (or a first operating system) into a rendering task (e.g., a target executable file) supported in a second system environment (or a second operating system). In this embodiment, the first system environment is distinct from the second system environment, and the physical GPU device 1 (i.e., the corresponding task processing device) supports the second system environment.

[0139] In the embodiment of the present application, the first system environment may refer to a Linux system environment, and the second system environment may refer to a Windows system environment. In other embodiments, the first system environment and the second system environment may also be other system environments.

[0140] The rendering applications 31 include OpenGL applications, Vulkan applications, Direct3D applications, etc. For example, since Direct3D applications have multiple versions and the embodiment of the present application has a conversion function, the embodiment of the present application can enable the task processing system to support conversion between various versions of Direct3D applications and OpenGL applications and / or Vulkan applications.

[0141] 11 , the embodiment of the present application can convert the rendering task of a Direct3D application into a rendering task of an OpenGL application or a Vulkan application. Converting the rendering task of a Direct3D application into a rendering task of an OpenGL application or a Vulkan application includes the following steps.

[0142] (1) The applications of various versions, such as Direct3D_9, Direct3D_10, and Direct3D_11, can be converted into OpenGL related calls.

[0143] (2) Convert the Direct3D_12 application into a Vulkan application rendering task for calling.

[0144] (3) Convert Direct3D coordinate system and clipping space.

[0145] (4) Able to implement Direct3D viewport conversion.

[0146] (5) Convert the high-level shader language HLSL into the shader language GLSL in OpenGL.

[0147] (6) Convert the shading language DXIL into the high-level intermediate language SPIRV in Vulkan applications.

[0148] The embodiment of the present application forms the conversion logic of 3D functions through the cooperation of the rendering processor set in the user-mode driver and the adapter set in the virtual GPU device, thereby achieving compatibility between various applications in different system environments and maximizing the rendering support for various system environments, for example: running Direct3D applications in the Linux system environment.

[0149] As shown in FIG11 , the kernel-mode driver includes a command buffer and a data buffer.

[0150] The command buffer is used to cache the target commands obtained from the target processor and schedule the pending command queue; wherein the pending command queue is composed of the cached target commands.

[0151] Data cache area, used to cache the data of the target task.

[0152] The kernel-mode driver can be set in the user-mode driver of the client virtual machine to: (1) allocate a command buffer and data space (i.e., the above-mentioned data buffer); (2) obtain commands and data submitted by the user-mode driver; (3) submit commands and data to the virtual GPU device; (4) process frame synchronization information; and (5) schedule a pending command queue consisting of pending commands in the command buffer.

[0153] The adapter 21 on the virtual GPU device 2 is described below with reference to FIG. 8 and FIG. 9 .

[0154] The adapter 21 can be set in the virtual GPU device 2 to process various tasks issued by the rendering processor 411, video processor 412 and parallel computing processor 413 in the client virtual machine, and manage the processing process of such tasks and corresponding physical resources (for example, the resources of the physical GPU device 1) (for example, scheduling physical GPU resources).

[0155] The adapter 21 can be configured to perform internal processing after receiving task data sent by the rendering processor 411, video processor 412 and parallel computing processor 413 in the client virtual machine. The specific process of the internal processing is shown in Figures 10 and 11 and will not be repeated here.

[0156] The adapter 21 may also be configured to calculate and allocate GPU resources according to the received tasks through the resource scheduling module 211. The GPU resources include physical GPU resources and virtual GPU resources.

[0157] The adapter 21 can also be configured to analyze the processing capabilities of the physical GPU device 1, and determine the capability values ​​of the resource scheduling module 211 based on the video memory size, number of stream processors, number of rasterization engines, etc. of the physical GPU device 1, such as: task queue size, maximum video frame rate (FramesPerSecond, FPS) supported, etc.

[0158] The adapter 21 may also be configured to encapsulate a rendering scheduling unit 221 , a video scheduling unit 222 , and a parallel computing scheduling unit 223 , so that each scheduling unit processes different types of tasks.

[0159] The adapter 21 can also be configured to assign different types of tasks to different functional logic units (i.e., the above-mentioned rendering scheduling unit 221, computing scheduling unit or video scheduling unit 222), and the assigned scheduling unit is used to abstractly encapsulate the logic of the physical GPU device 1.

[0160] The adapter 21 can also be configured to parse specific rendering commands in the rendering scheduling unit 221, and control the physical GPU device 1 to allocate rendering resources, such as texture, buffer and other resources through the functional interface provided by the driver of the physical GPU device 1, and then allow the physical GPU device 1 to process the specified rendering task.

[0161] The adapter 21 can also be configured to parse the video acceleration command in the video scheduling unit 222, and send a processing request corresponding to the video task to the physical GPU device 1 through the data cache module to allow the physical GPU device 1 to allocate physical GPU resources corresponding to the video task, such as video buffer, etc.

[0162] The adapter 21 may also be configured to parse the parallel computing command in the parallel computing scheduling unit 223 , encapsulate the instruction corresponding to the physical GPU resource requirement information, and send the instruction to the physical GPU device 1 .

[0163] The adapter 21 can also be configured to periodically measure the task processing load in each scheduling unit and dispatch tasks to each scheduling unit according to the set strategy. For example, if the number of vertices per unit time exceeds a preset threshold, the rendering task is suspended to avoid task congestion.

[0164] The adapter 21 may also be configured to respond to the virtual GPU device 2 interrupting the task and / or postponing the scheduler, and write back the processing results obtained by each scheduling unit by calling the physical GPU device 1 to process the target task.

[0165] The adapter in the embodiment of the present application can adapt to the processors of various functions, cache data / commands, manage rendering and computing status, manage physical GPU resources, and allocate physical GPU resources for rendering tasks, video tasks, parallel computing tasks, etc., to achieve reasonable scheduling of physical GPU resources.

[0166] In a possible implementation, in the adapter 21 , the target scheduling unit 2111 may include at least one of a rendering scheduling unit 221 , a video scheduling unit 222 , or a parallel computing scheduling unit 223 .

[0167] The rendering scheduling unit 221 may be configured to parse the rendering requirement information of the rendering task from the rendering command.

[0168] The video scheduling unit 222 may be configured to parse the video acceleration command to obtain video requirement information of the video task.

[0169] The parallel computing scheduling unit 223 may be configured to parse the computing requirement information of the parallel computing task from the parallel computing command.

[0170] Compared to related art virtual GPU technologies that rely on binding fixed physical GPU resources, the virtual GPU technology in this embodiment can virtualize the full functionality of the GPU. Specifically, this embodiment can virtualize rendering, parallel computing, and video acceleration processing, independent of binding fixed physical GPU resources. By utilizing the rendering processor, parallel computing processor, and video processor configured in the user-mode driver of the client virtual machine, as well as the rendering scheduling unit, video scheduling unit, and parallel computing scheduling unit in the adapter of the virtual GPU device, the virtual GPU device abstractly manages the physical GPU device (including GPU instruction conversion, data format conversion, and resource management). While fully supporting the virtual GPU device, it also supports the free allocation and dynamic scaling of physical GPU resources.

[0171] As shown in FIG. 9 and FIG. 11 , the adapter 21 may further include a command cache module 212 and a data cache module 213 .

[0172] The command cache module is used to cache the target commands obtained from the command cache area. The data cache module is used to cache the target task data obtained from the data cache area, as well as the processing feedback results of the target task.

[0173] The command buffer module is also called a queue buffer module, and the data buffer module is also called a data buffer module.

[0174] On the basis that the virtual GPU device includes an adapter, the virtual GPU device in the embodiment of the present application can implement the following functions.

[0175] (1) The virtual GPU device is used to allocate physical GPU resources; wherein, the physical GPU device corresponding to the virtual GPU device has a memory vram for managing the allocation of physical GPU resources, which can be adaptively used according to different configurations and functions.

[0176] (2) After command parsing, the virtual GPU device allocates rendering and / or computing resources according to the resource requirements of the target task corresponding to the target command. The resources include physical GPU resources and virtual GPU resources.

[0177] (3) The virtual GPU device is also used to process the command list in the command cache module. The command list is a command stream in the command cache module and is used to cache commands for rendering processing, video acceleration processing, and parallel computing processing.

[0178] (4) The virtual GPU device is also used to process frame synchronization information, which may include video acceleration processing and rendering processing.

[0179] (5) Virtual GPU devices are also used to manage rendering processes, parallel computing processes, and video processing processes. Specifically, virtual GPU devices are used to manage rendering processes, such as drawcall processing status, resource data (such as textures, buffers, etc.) processing progress, and rendering result writeback. Virtual GPU devices are also used to manage parallel computing processes, such as kernel function compute shader status detection, resource settings, and parallel computing task distribution. Virtual GPU devices are also used to manage video processing processes, such as parameter preparation and cache management corresponding to formats such as h264 / h265.

[0180] (6) Virtual GPU devices are also used to manage physical GPU resources. Specifically, this embodiment can allocate rendering computing resources, such as rendering textures and buffers, from physical GPU resources based on the needs of the various functional scheduling modules.

[0181] The scheduling process is analyzed as follows in conjunction with FIG12 .

[0182] Step 121: The virtual GPU device receives commands and data from the command buffer and data buffer provided by the kernel-mode driver.

[0183] Step 122: Receive the video acceleration command and frame information from the data buffer provided by the kernel-mode driver.

[0184] Step 123: The virtual GPU device may perform command parsing through a built-in resource scheduling module.

[0185] Step 124: After the command parsing is completed, the virtual GPU device calculates resource allocation.

[0186] When allocating computing resources, physical GPU resources can be allocated according to the command requirement information corresponding to the target task.

[0187] When the command requirement information is rendering requirement information, the GPU resources required for processing the rendering requirement information are calculated, and corresponding physical GPU resources are provided as GPU rendering resources.

[0188] When the command requirement information is video requirement information, the GPU resources required for processing the video requirement information are calculated, and the corresponding physical GPU resources are provided as GPU video resources.

[0189] When the command requirement information is computing requirement information, the GPU resources required for processing the computing requirement information are calculated, and corresponding physical GPU resources are provided as GPU computing resources.

[0190] Step 125: Coordinate and adapt GPU resources based on the calculated resource allocation, submit corresponding processing commands, and collect processing results.

[0191] Specifically, 1) the rendering scheduling unit in the virtual GPU device is used to coordinate and adapt GPU rendering resources, submit rendering commands, and collect processing results; 2) the video scheduling unit in the virtual GPU device is used to coordinate and adapt GPU video processing resources, submit rendering commands, and collect processing results; 3) the parallel computing scheduling in the virtual GPU device is used to coordinate and adapt GPU computing resources, submit computing commands, and collect processing results.

[0192] Step 126: Write the processing result to the data cache module.

[0193] The above steps 124 to 125 together complete the abstraction of physical GPU resources to achieve dynamic management of physical GPU resources.

[0194] The embodiments of the present application have the following advantages: (1) This embodiment can improve the compatibility of the operating system software running in the client virtual machine and can support various 3D applications, such as: 3D modeling, animation production, simulation, VR / AR, video playback, etc.; (2) This embodiment can improve the graphics operation performance of the operating system in the client virtual machine, and under the same computing resources, the smoothness of ordinary graphics operations is improved by more than 60%; (3) This embodiment can reduce the construction cost of the related virtual GPU technology solution and achieve cost savings of more than 300%.

[0195] In addition, the functions included in the embodiments of the present application are not limited to: virtualization of ordinary GPU rendering, video acceleration, parallel computing acceleration and other functions, and it can expand more other computing acceleration support. For example, this embodiment can expand virtualization support for AI computing acceleration dedicated chips.

[0196] Based on the embodiments of FIG. 8 to FIG. 12 above, the technical solution of the present application will be described in more detail below in conjunction with specific embodiments.

[0197] FIG13 is a schematic diagram of the structure of a virtual GPU architecture provided in an embodiment of the present application. The virtual GPU architecture can be understood as another architecture of a task processing system.

[0198] This virtual GPU architecture is equipped with a virtual GPU device and a physical GPU device on the host. Different client virtual machines can include Linux virtual machines and Windows virtual machines. Among them, Linux applications are set up on the Linux virtual machine. Each software initiates corresponding tasks to the user-state driver mesa2 within the runtime OpenGL runtime, OpenCL runtime and Libva runtime. The user-state driver mesa2 is deployed with processors corresponding to each function (including rendering processor, video processor and parallel computing processor), and sends corresponding commands and data to the virtual GPU device through the kernel-state driver drm2.

[0199] Among them, Windows applications are set up on the Windows virtual machine. Each software initiates the corresponding task to the user-state driver wddm umd within the runtime OpenGL runtime, OpenCL runtime and Dxva runtime. The user-state driver wddm umd is deployed with processors corresponding to each function (including rendering processor, video processor and parallel computing processor), and sends the corresponding commands and data to the virtual GPU device through the kernel-state driver wddm kmd.

[0200] The virtual GPU device is equipped with an adapter, which in turn hosts dispatching units corresponding to various functions (including a rendering unit, a video unit, and a parallel computing unit). The host also hosts the user-mode driver mesa1 and the kernel-mode driver drm1. These three drivers, working in sequence, enable resource access to the physical GPU device.

[0201] The embodiments of the present application provide the following solutions in the process of implementing on-demand call of physical GPU resources:

[0202] (1) This embodiment is based on the collaborative work between the driver (for example, the user-mode driver umd, the kernel-mode driver drm, etc.) and the virtual GPU device to enable the virtual GPU device to call the physical GPU resources. The above collaborative process is shown in Figure 11 provided in the above embodiment and will not be repeated here.

[0203] (2) Rendering adaptation, video acceleration adaptation, and parallel computing adaptation are implemented through the scheduling units provided by the adapter to implement rendering, video acceleration, parallel computing acceleration, and other functions within the virtual machine, thereby realizing various logical functions of the virtual GPU device.

[0204] (3) The client virtual machine includes a specific functional unit driven by a virtual GPU device (see the functional module in FIG12 ).

[0205] (4) The virtual GPU device provides virtual GPU function support for audio and video encoding / decoding and parallel computing acceleration.

[0206] (5) This embodiment implements support for 3D rendering applications such as OpenGL applications, Direct3D applications, and Vulkan applications in the system environment of the client virtual machine. Functional docking can be completed by a function converter, and the input of the function converter is various 3D applications to be converted.

[0207] (6) This embodiment implements virtualization of the virtual GPU device on the host, and can complete the abstract management of the hardware resources of the physical GPU device.

[0208] (7) This embodiment can realize the calling / conversion between applications through the renderer processor, video processor, parallel computing processor, etc. deployed in the user mode driver of the Windows virtual machine, so that the virtual GPU device can support both Windows virtual machines and Linux virtual machines.

[0209] (8) This embodiment supports on-demand use of physical GPU resources, thereby improving the utilization rate of physical GPU resources.

[0210] In summary, this virtual GPU architecture can provide a universal virtual GPU solution, and it can realize the various functions of virtual GPU devices based on virtualization technology. The advantage is that it can be separated from the binding of physical GPU devices and can run independently, supporting the flexible allocation of physical GPU resources.

[0211] FIG14 is a flowchart illustrating a task processing method provided in an embodiment of the present application. The method is applied to the task processing system shown in FIG8 ; the task processing system includes: a virtual GPU device located on a host machine; a target application and a virtual GPU driver deployed on a client virtual machine; the virtual GPU driver virtual program includes a user-mode driver and a kernel-mode driver; the user-mode driver includes multiple processors; and the virtual GPU device includes an adapter, which includes a resource scheduling module, which includes multiple scheduling units.

[0212] As shown in FIG14 , the task processing method includes the following steps:

[0213] S1. A target processor in a user-mode driver of a client virtual machine generates a target command in response to a processing request of a target task, and sends the target command to an adapter in a virtual GPU device via a kernel-mode driver in the client virtual machine. The target processor is a processor among multiple processors that matches a target application; the target task is a GPU task initiated by the target application; the target command is a command corresponding to the target task, and the target command includes physical GPU resource requirement information and / or CPU resource requirement information required to process the target task.

[0214] S2. Allocate the target command to a target scheduling unit through the resource scheduling module, wherein the target scheduling unit is a scheduling unit that matches the target application among the multiple scheduling units.

[0215] S3. The target scheduling unit parses command requirement information from the target command, and allocates physical GPU resource availability information and / or CPU resource availability information according to the command requirement information.

[0216] S4. When the physical GPU resource available information and / or the CPU resource available information is allocated, the target task is processed by the physical GPU device and / or the CPU device to obtain a processing feedback result.

[0217] The physical GPU resource availability information is used to indicate the available GPU resources, and the CPU resource availability information is used to indicate the available CPU resources. The target task can be a GPU task, which can include rendering tasks, video acceleration tasks, and parallel computing tasks. This embodiment provides a preset resource utilization policy for the target task. This preset resource utilization policy can prioritize the use of physical GPU devices.

[0218] If the available GPU resources are higher than the resources required to process the target task, the physical GPU device is used for processing.

[0219] If the available GPU resources are higher than the preset threshold but lower than the resources required to process the task, the physical GPU device and the CPU device are used together for processing.

[0220] If the GPU resources available are lower than the preset threshold, the CPU device is used for processing.

[0221] Optionally, the target application includes: at least one of a rendering application, a video application and a parallel computing application; the target processor includes: at least one of a rendering processor, a video processor and a parallel computing processor; the target scheduling unit includes: at least one of a rendering scheduling unit, a video scheduling unit and a parallel computing scheduling unit; the target command includes: at least one of a rendering command, a video acceleration command and a parallel computing command.

[0222] Optionally, S1, generating a target command by a target processor provided in a user-mode driver of the client virtual machine in response to a processing request of a target task, and sending the target command to an adapter provided in a virtual GPU device by a kernel-mode driver provided in the client virtual machine, includes at least one of the following:

[0223] The rendering processor generates a rendering command in response to a processing request for a rendering task initiated by the rendering application, and sends the rendering command to the adapter through a kernel-mode driver.

[0224] The video processor generates a video acceleration command in response to a processing request of a video task initiated by a video application, and sends the video acceleration command to the adapter through a kernel-mode driver.

[0225] The parallel computing processor generates a parallel computing command in response to a processing request for a parallel computing task initiated by a parallel computing application, and sends the parallel computing command to the adapter through a kernel-mode driver.

[0226] Optionally, S3: parsing command requirement information from the target command by the target scheduling unit, including at least one of the following:

[0227] The rendering scheduling unit parses the rendering command to obtain the rendering requirement information of the rendering task.

[0228] The video scheduling unit parses the video acceleration command to obtain the video requirement information of the video task.

[0229] The parallel computing scheduling unit parses the computing requirement information of the parallel computing task from the parallel computing command.

[0230] Optionally, the method further comprises:

[0231] S5. Converting, by a rendering processor, a target task initiated by the rendering application in the first system environment into a target executable file in the second system environment; wherein the first system environment is different from the second system environment, and the physical GPU device supports the second system environment.

[0232] Optionally, the kernel-mode driver includes a command buffer and a data buffer; the method further includes:

[0233] S6. Cache the target command obtained from the target processor through the command buffer area, and schedule a queue of pending commands; wherein the queue of pending commands is composed of the cached target commands.

[0234] S7. Cache the target task data in the data buffer area.

[0235] Optionally, the adapter further includes: a command cache module and a data cache module; and the method further includes:

[0236] S8. Cache the target command obtained from the command buffer area through the command buffer module.

[0237] S9. Cache the target task data obtained from the data cache area and the processing feedback result of the target task through the data cache module.

[0238] Optionally, the virtual GPU device is further configured to perform any of the following:

[0239] Allocate physical GPU resources; wherein, the physical GPU device corresponding to the virtual GPU device has a memory vram for managing the allocation of physical GPU resources;

[0240] After command parsing, rendering and / or computing resources are allocated according to the resource requirements of the target task corresponding to the target command. Resources include physical GPU resources and virtual GPU resources.

[0241] Process the command list in the command cache module. The command list is the command stream in the command cache module, which is used to cache rendering commands, video acceleration commands, and parallel computing commands;

[0242] Processing frame synchronization information, where the frame synchronization information is used to implement video acceleration processing and / or 3D rendering processing;

[0243] Manage the state of target processing operations, including rendering, parallel computing, and video processing;

[0244] Manage physical GPU resources.

[0245] The implementation principle and technical effects of the task processing method provided in this embodiment are similar to those of the above-mentioned system embodiment and will not be described in detail here.

[0246] In one embodiment, the user information and data involved in this application (including but not limited to data used for analysis, stored data, displayed data, etc.) may all be information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0247] Each exemplary embodiment of the present application further provides a task processing method. As shown in FIG15 , the method includes the following steps.

[0248] In step S21 , the client virtual machine generates a target command in response to a processing request of a target task.

[0249] In step S22 , the client virtual machine sends the target command to an adapter provided in a virtual graphics processor (GPU) device of the host machine.

[0250] [Corrected 24.02.2025 according to Rule 91] Step S23, the adapter parses the command requirement information corresponding to the target command from the target command, and confirms the available information of the task processing resources of the target task based on the command requirement information.

[0251] Step S24 : in response to the confirmation of the task processing resource availability information, the adapter allows the corresponding task processing device to process the target task based on the task processing resource availability information to obtain a processing result.

[0252] The steps of the above method and its other variations are consistent with the embodiments of the aforementioned task processing method and will not be described in detail herein.

[0253] Each exemplary embodiment of the present application further provides a task processing device, as shown in FIG16 , which includes a client virtual machine and an adapter disposed in a host machine.

[0254] The client virtual machine is configured to generate a target command in response to a processing request of a target task, and is configured to send the target command to an adapter provided in a virtual graphics processor (GPU) device of the host machine.

[0255] The adapter is configured to parse command requirement information from the target command and confirm the task processing resource availability information of the target task based on the command requirement information; and is configured to allow the corresponding task processing device to process the target task based on the task processing resource availability information in response to the confirmation of the task processing resource availability information to obtain a processing result.

[0256] The functions of the client virtual machine and adapter of the above-mentioned device and other variants thereof are consistent with those of the embodiment of the aforementioned task processing method and will not be described in detail herein.

[0257] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0258] Figure 17 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device includes a receiver 60, a transmitter 61, at least one processor 62, and a memory 63. The electronic device composed of the above components can be used to implement the above-mentioned specific embodiments of the present application, which will not be repeated here.

[0259] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, each step of the method in the above embodiment is implemented.

[0260] An embodiment of the present application further provides a computer program product, including a computer program, which implements each step of the method in the above embodiment when executed by a processor.

[0261] Various embodiments of the systems and techniques described above in this application can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0262] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or electronic device.

[0263] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable storage medium can be a machine-readable signal medium or a machine-readable storage medium. A computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. A more specific example of a computer-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0264] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0265] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as data electronics), or a computing system that includes middleware components (e.g., application electronics), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0266] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved. This is not a limitation herein.

[0267] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this application shall be included within the scope of protection of this application.

Claims

1. [Corrected 24.02.2025 in accordance with Article 91] A method of processing a task, comprising: The client virtual machine generates a target command in response to the processing request of the target task; The client virtual machine sends the target command to an adapter set in a virtual graphics processor GPU device of the host machine; Parsing command requirement information corresponding to the target command from the target command through the adapter, and confirming available task processing resource information of the target task according to the command requirement information; as well as In response to confirmation of the task processing resource availability information, the adapter allows the corresponding task processing device to process the target task based on the task processing resource availability information to obtain a processing result.

2. The method according to claim 1, wherein: The adapter allows the corresponding task processing device to process the target task based on the task processing resource availability information, including: The adapter instructs the corresponding task processing device to allocate task processing resources for the target task based on the task processing resource availability information.

3. The method according to claim 1 or 2, wherein: Before the adapter allows the corresponding task processing device to process the target task based on the task processing resource availability information, the method further includes: The adapter obtains the processing load status of the corresponding task processing device; and In response to the processing load status being greater than or equal to a first threshold, the adapter suspends allowing a corresponding task processing device to process the target task based on the task processing resource availability information.

4. The method according to claim 3, wherein: In response to the processing load state being greater than or equal to a first threshold, the adapter suspending the corresponding task processing device from processing the target task based on the task processing resource availability information, including: In response to the number of vertices of resources to be rendered by the corresponding task processing device within a preset time exceeding a second threshold, the adapter suspends the corresponding task processing device from processing the target task based on the task processing resource availability information.

5. The method according to any one of claims 1 to 4, wherein: The target command includes at least one of a rendering command, a video acceleration command or a parallel computing command.

6. The method according to any one of claims 1 to 5, wherein: The client virtual machine generates the target command in response to the processing request of the target task, including: The target command is generated by setting a target processor in the user mode driver of the client virtual machine in response to a processing request of the target task, wherein the target processor is a processor matching the target task among one or more processors in the client virtual machine.

7. The method according to claim 6, wherein: The step of parsing the command requirement information from the target command by the adapter includes: Allocating the target command to a target scheduling unit of the resource scheduling module through the resource scheduling module of the adapter, wherein the target scheduling unit is a scheduling unit corresponding to the target task among one or more scheduling units of the resource scheduling module; and The command requirement information is parsed from the target command by the target scheduling unit.

8. The method according to claim 6 or 7, wherein: The processing request of the target task comes from a target application; The target application includes: at least one of a rendering application, a video application, or a parallel computing application; The one or more processors include: at least one of a rendering processor, a video processor, or a parallel computing processor; and The one or more scheduling units include: at least one of a rendering scheduling unit, a video scheduling unit, and a parallel computing scheduling unit.

9. The method according to claim 8, wherein: The target processor generates the target command in response to the processing request of the target task, and the client virtual machine sends the target command to the adapter set in the GPU device of the host machine, including: If the target application is the rendering application, the rendering processor generates the rendering command in response to the processing request of the target task initiated by the rendering application, and the client virtual machine sends the rendering command to the adapter; If the target application is the video application, generating the video acceleration command by the video processor in response to the processing request of the target task initiated by the video application, and the client virtual machine sends the video acceleration command to the adapter; and If the target application is the parallel computing application, the parallel computing processor generates the parallel computing command in response to the processing request of the target task initiated by the parallel computing application, and the client virtual machine sends the parallel computing command to the adapter.

10. The method according to claim 8 or 9, wherein: The step of parsing the command requirement information from the target command by the target scheduling unit includes: If the target command is the rendering command, parsing rendering requirement information of the rendering task from the rendering command through the rendering scheduling unit; If the target command is the video acceleration command, parsing the video requirement information of the video task from the video acceleration command by the video scheduling unit; and If the command is the parallel computing command, the parallel computing scheduling unit parses the parallel computing command to obtain computing requirement information of the parallel computing task.

11. The method according to any one of claims 8 to 10, wherein: The method further comprises: The rendering processor converts the target task initiated by the rendering application in the first system environment into a rendering task supported in the second system environment; wherein the first system environment is different from the second system environment, and the corresponding task processing device supports the second system environment.

12. The method according to claims 1 to 11, wherein: The client virtual machine sends the target command to the adapter set in the GPU device of the host machine, including: The client virtual machine sends the target command to the adapter set in the GPU device of the host machine through a kernel mode driver set in the client virtual machine.

13. The method according to claim 12, wherein: The kernel mode driver includes at least a command buffer and a data buffer, and the method further includes: Acquire the target command from the target processor through the command buffer cache; Scheduling a pending command queue through the command buffer, wherein the target command is buffered in the pending command queue; and The data of the target task is cached through the data cache area.

14. The method according to claim 13, wherein: The adapter further includes: a command cache module and a data cache module, and the method further includes: caching the target command obtained from the command buffer area by the command buffer module; and The data of the target task acquired from the data cache area and the processing result of the target task are cached by the data cache module.

15. The method according to any one of claims 1 to 14, wherein: The task processing resource availability information is confirmed based on the physical GPU resource availability information and the CPU resource availability information.

16. The method according to claim 15, wherein: The step of confirming the available information of the task processing resources of the target task according to the command requirement information includes: If the GPU resource available information is greater than or equal to the command requirement information, confirming that the GPU resource available information is the task processing resource available information; If the GPU resource available information is less than the command requirement information and is greater than or equal to a third threshold, confirming that the GPU resource available information and the CPU resource available information are the task processing resource available information, wherein the third threshold is less than the command requirement information; and If the GPU resource availability information is less than the third threshold, the CPU resource availability information is confirmed as the task processing resource availability information.

17. The method according to any one of claims 1 to 16, wherein: The corresponding rendering device includes at least one of a physical GPU device and a CPU device.

18. A task processing device, comprising: The client virtual machine is configured to generate a target command in response to a processing request of a target task, and is configured to send the target command to an adapter provided in a virtual graphics processor GPU device of the host machine; as well as The adapter is configured to parse command requirement information from the target command, and confirm task processing resource availability information of the target task according to the command requirement information; The system is configured to allow the corresponding task processing device to process the target task based on the task processing resource availability information in response to confirmation of the task processing resource availability information to obtain a processing result.

19. An electronic device comprising: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the task processing method according to any one of claims 1 to 17.

20. A computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are used to implement the task processing method according to any one of claims 1 to 17 when executed by a processor.

Images