Interface interaction control method, computer system, equipment and storage medium

Abstract

The embodiment of the invention discloses an interface interaction control method, a computer system, equipment and a computer storage medium. According to an interface type of a target virtual interface indicated by the creation request, creating a target virtual computing power device corresponding to the interface type in a target virtual environment, responding to a deep learning request of the target virtual environment through the target virtual computing power device, and forwarding the request to a host machine, and scheduling a physical computing power device corresponding to the interface type to respond to the request, and returning a processing result of the request to the target virtual environment through a target virtual computing power device. According to the method, virtualization is carried out from a software interaction level instead of virtualization of computing power hardware, the scheme is completely realized based on software, and a physical GPU is not required to have any special virtualization hardware function, so that almost all mainstream and even old models of GPUs on the market and accelerators such as NPUs with different architectures can be managed. And the user does not need to pay for expensive virtualization dedicated hardware and software license, so that the virtualization deployment cost is reduced.

Description

Technical Field

[0001] This application relates to the field of software management, specifically to an interface interaction control method, a computer system, a device, and a computer storage medium. Background Technology

[0002] Hardware virtualization is currently widely used in AI computing scenarios. Represented by vGPU technology, it's a technique that relies on the built-in hardware functions of a physical GPU (such as SR-IOV) or dedicated firmware support to partition the computing power and memory resources of a single physical GPU at the hardware level, generating multiple completely isolated and fully functional virtual GPU instances. Each virtual GPU instance can be directly assigned to a virtual machine, and the operating system and applications within the virtual machine can use it as if it were an independent physical GPU, offering good performance isolation and security. However, this technical approach has significant drawbacks.

[0003] First, the cost is extremely high, requiring the purchase of specific high-end hardware models that support virtualization capabilities, as well as proprietary software licenses from the vendors. This means that the software licensing costs for large-scale deployments can far exceed the hardware costs themselves, significantly increasing the total cost of ownership. Second, this approach leads to vendor lock-in and technology lock-in. The virtual GPU's management program, driver stack, and licensing server are all provided and controlled entirely by the hardware vendor, forming a closed technological ecosystem. Once deployed, the user's virtualization environment's infrastructure, operational processes, and even business applications are deeply coupled with the specific vendor's products, making it difficult to migrate or introduce hardware from other brands, thus losing the flexibility and scalability of technology choices.

[0004] Therefore, although hardware virtualization offers good performance and isolation, its high deployment costs, closed ecosystem binding, and limited resource elasticity limit its widespread application in large-scale AI heterogeneous computing power clusters. Summary of the Invention

[0005] This application provides an interface interaction control method, computer system, device, and computer storage medium that performs virtualization at the software interaction level. By creating a target virtual computing power device, it enables interaction between different virtual machines and the host machine's physical computing power hardware, thereby reducing virtualization deployment costs and improving the scalability of virtualization deployment.

[0006] The first aspect of this application provides an interface interaction control method, which is applied to a computer system, the computer system including a host machine, the computing resources of which are used to support the operation of a virtual environment; the method includes:

[0007] In response to a request to create a target virtual computing device, a target virtual computing device corresponding to the interface type of the target virtual interface indicated in the creation request is created in the target virtual environment.

[0008] The target virtual computing device responds to the deep learning request of the target virtual environment and forwards the deep learning request to the host machine;

[0009] The system schedules the physical computing device corresponding to the interface type to respond to the deep learning request, and returns the processing result of the deep learning request to the target virtual environment through the target virtual computing device.

[0010] A second aspect of this application provides a computer system, the computer system including a host machine, the computing resources of which are used to support the operation of a virtual environment; the computer system includes:

[0011] A creation unit is configured to respond to a creation request for a target virtual computing power device and, based on the interface type of the target virtual interface indicated by the creation request, create a target virtual computing power device in the target virtual environment corresponding to the interface type.

[0012] The sending unit is configured to respond to the deep learning request of the target virtual environment through the target virtual computing power device, and forward the deep learning request to the host machine;

[0013] The scheduling unit is used to schedule the physical computing power device corresponding to the interface type to respond to the deep learning request, and return the processing result of the deep learning request to the target virtual environment through the target virtual computing power device.

[0014] A third aspect of this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method of the first aspect described above.

[0015] A fourth aspect of this application provides a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0016] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0017] Virtualization is implemented at the software interaction level, rather than at the computing hardware level. It achieves interaction between different virtual machines and the host machine's physical computing hardware by creating target virtual computing devices. Because the solution is entirely software-based, it does not require the physical GPU to have any special virtualization hardware functions. Therefore, it can manage almost all mainstream and even older GPUs on the market, as well as accelerators such as NPUs with different architectures. Users do not need to pay for expensive virtualization-specific hardware and software licenses, reducing virtualization deployment costs. At the same time, applications use stable virtual API interfaces, rather than native drivers from a particular vendor, thus freeing applications from specific hardware ecosystems, avoiding dependence on a single hardware vendor, and improving the scalability and flexibility of virtualization deployment.

[0018] When it is necessary to upgrade, expand or replace physical computing power equipment, only the hardware adaptation and computing power pool update need to be completed on the host side. The computing services in the target virtual environment can seamlessly call the new physical computing power resources without modifying any code or recompiling. This greatly reduces the operation and maintenance costs and business migration risks of heterogeneous computing power clusters and improves the resource elasticity of computing power hardware. Attached Figure Description

[0019] Figure 1 This is an exemplary schematic diagram of a heterogeneous computing architecture for AI computing scenarios in related solutions;

[0020] Figure 2 This is another exemplary schematic diagram of a heterogeneous computing architecture for AI computing scenarios in related solutions;

[0021] Figure 3 This is a schematic diagram of an exemplary scenario architecture according to an embodiment of this application;

[0022] Figure 4 This is a flowchart illustrating the interface interaction control method in an embodiment of this application.

[0023] Figure 5 This is a schematic diagram of the architecture of an exemplary target virtual computing device in an embodiment of this application;

[0024] Figure 6 This is an exemplary schematic diagram illustrating the overall operation process of creating a resource pool, allocating virtual devices, and distributing configurations through a management interface or API in this application embodiment.

[0025] Figure 7 This is a schematic diagram of an exemplary scenario of the interface interaction control method in the embodiments of this application;

[0026] Figure 8 This is a schematic diagram of a computer device in an embodiment of this application. Detailed Implementation

[0027] This application provides an interface interaction control method, computer system, device, and computer storage medium that performs virtualization at the software interaction level. By creating a target virtual computing power device, it enables interaction between different virtual machines and the host machine's physical computing power hardware, thereby reducing virtualization deployment costs and improving the scalability of virtualization deployment.

[0028] Currently, mainstream heterogeneous accelerators, such as NVIDIA GPUs, are quite expensive. How to efficiently utilize heterogeneous accelerators has been a topic of ongoing exploration in the industry. Currently, due to the lack of efficient pooling solutions, the traditional model of using accelerators independently suffers from the following problems:

[0029] The waste of resources is due to the fact that different models have different resource requirements for accelerators, and it is difficult to produce a hardware model according to the needs of each type of model. Therefore, the hardware specifications are difficult to match the different resource requirements of various models, resulting in some hardware resources being unused and wasted.

[0030] Complex development and deployment or vendor lock-in: Due to the lack of heterogeneous computing power support, the platform requires modification of applications to adapt to accelerator hardware. This leads to complex application development and deployment or lock-in to a certain hardware vendor, making it difficult to expand hardware computing power.

[0031] Applications are restricted to single physical machines; they must be deployed on a specific server and can only use the accelerator on the current host.

[0032] While some hardware manufacturers support hardware virtualization solutions, there are model limitations, and the cost of their virtualization-related serial numbers is very high, which can easily lead to the solution being tied to the hardware manufacturer and difficult to expand.

[0033] In terms of technical principles, the normal driving method for heterogeneous computing power software pooling in current AI computing scenarios is for the APP to call the underlying library, such as... Figure 1 As shown, the underlying library's application calls the kernel driver, and the kernel driver calls the hardware. The underlying library and driver access each other through device files, therefore, the key to interception is the device file. The kernel interception module actually calls the GPU driver, and then parses the response upon return.

[0034] However, this scenario is suitable for container applications but not for virtual machines because containers share the host operating system's kernel, while virtual machines typically run independent operating system kernels. In virtual machines, since each virtual machine has its own kernel, GPU driver interception can potentially involve more complexity and performance overhead. Kernel sharing in containers can simplify GPU driver management and performance optimization, but the downside is that multiple applications share the same operating system, resulting in compromised isolation.

[0035] In virtual machine usage scenarios, besides hardware virtualization solutions, there are also paravirtualization and full virtualization solutions. Full virtualization has excessive performance overhead and is generally unsuitable for widespread commercial use. Paravirtualization solutions mainly consist of two parts: drivers that run within the virtual machine but interact with the host machine programs; and pooling / virtualization components running on the host machine, which are the physical hardware responsible for receiving and forwarding requests. For example, its data flow is illustrated as follows... Figure 2 As shown, this demonstrates the complete process of virtualizing OpenGL drawing instructions and forwarding them to the host machine's physical GPU for execution when running 3D graphics applications (such as CAD and 3D games) in a virtual machine. It is a typical "command stream virtualization" scheme. The following is a detailed explanation of this data flow:

[0036] Within the virtual machine (Guest), the Guest App can be an application that requires 3D graphics acceleration (such as a 3D design software). An OpenGL call refers to an application issuing drawing instructions (such as drawing triangles or applying textures) through the standard OpenGL graphics API.

[0037] Mesa 3D driver refers to an open-source graphics driver that implements the OpenGL standard and runs within a virtual machine. It is responsible for receiving OpenGL calls.

[0038] Generating Virgl command streams means that the Mesa driver does not directly operate the virtual GPU hardware, but instead converts (encodes) OpenGL instructions into an intermediate command stream format called "Virgl," which is independent of the specific GPU hardware.

[0039] VirtIO GPU Front-end is a standardized virtual GPU device front-end (based on the VirtIO framework). It is responsible for sending the "Virgl command stream" to the host machine through the efficient VirtIO mechanism.

[0040] Within the host machine, the QEMU Back-end works in conjunction with the virtual machine front-end. It receives "Virgl command stream" packets from the front-end. Decoding the command stream refers to the initial processing of the command stream by the QEMU back-end.

[0041] The Virgl renderer is the core conversion component. It decodes and retranslates the generic "Virgl command stream" into native graphics API instructions (such as OpenGL or Vulkan) that the host machine's physical GPU can recognize.

[0042] The Host GPU API is used to call the host operating system's physical GPU driver (such as NVIDIA driver or AMD driver) to execute actual rendering commands.

[0043] Physical GPU rendering means that the final image is generated by physical GPU rendering. The rendering result (image frame) is usually sent back to the virtual machine through another channel (such as display virtualization) and presented to the user.

[0044] Based on the aforementioned computing scenarios, the most mature semi-virtualized vGPU in the industry is currently virtio-GPU, which is primarily based on 3D rendering. However, due to the lack of mature and universal standards and its high implementation complexity, there are currently no mature semi-virtualized devices for AI computing. Virtio, a semi-virtualization framework, provides an efficient standard communication protocol between virtual machines and the host machine, defining how the front-end driver and back-end implementation of virtual devices communicate.

[0045] Meanwhile, virtual GPUs also have several drawbacks. First, their cost is extremely high, requiring the purchase of specific high-end hardware models that support virtualization capabilities, as well as proprietary software licenses from the vendors. This means that the software licensing costs for large-scale deployments can far exceed the hardware itself, significantly increasing the total cost of ownership. Second, this approach leads to vendor lock-in and technology lock-in. The virtual GPU's management program, driver stack, and licensing server are all entirely provided and controlled by the hardware vendor, forming a closed technological ecosystem. Once deployed, the user's virtualization environment's infrastructure, operational processes, and even business applications are deeply coupled with the specific vendor's products, making it difficult to migrate or introduce hardware from other brands, thus losing the flexibility and scalability of technology choices.

[0046] To address the aforementioned technical problems, this application proposes an interface interaction control method, and an exemplary scenario architecture can be found below. Figure 3 This architecture diagram clearly illustrates the core system architecture and complete data flow of the embodiments of this application, which is divided into three layers from high to low: virtual machine (Guest), host machine, and physical heterogeneous computing power (Physical Hardware). The functions of each layer and core components are explained in detail below:

[0047] 1. Virtual machine layer;

[0048] This is the starting point for the operating environment and virtualization functions of AI applications.

[0049] Guest ML App: Represents an AI / machine learning application running in a virtual machine.

[0050] API calls: Applications issue computation requests through standard programming interfaces. The diagram clearly shows two main types: CUDA calls, which are geared towards the NVIDIA ecosystem; and OpenCL calls, which are geared towards the open, cross-platform ecosystem.

[0051] vAPI driver: This is the core of virtualization and optimization. The diagram shows two separate driver modules, including:

[0052] vCUDA driver: Specifically designed to hijack, parse, and optimize CUDA API calls;

[0053] vOpenCL driver: Specifically designed to hijack, parse, and optimize OpenCL API calls.

[0054] Generate command stream: The driver translates (or serializes) the optimized API request into a "command stream" format that is independent of the specific physical hardware.

[0055] VirtIO Front End: This is a standardized virtual I / O device (as shown in VirtIO-cuda and VirtIO-opencl in the figure). It acts as a data transmission channel and is responsible for sending the "command stream" to the host machine through the efficient VirtIO mechanism.

[0056] 2. Host layer;

[0057] This is the core and scheduling hub of virtualization management.

[0058] The QEMU backend works in pairs with the VirtIO frontend within the virtual machine. It receives "command stream" packets from the frontend.

[0059] The heterogeneous computing power pooling service is responsible for decoding the command stream, restoring the API semantics of the received general "command stream"; scheduling, i.e., determining which physical GPU / NPU will execute the task based on the computing power allocation strategy and quota; and API escaping, translating the decoded general commands into the native API calls corresponding to the target physical hardware (i.e., "Host computing power API" in the diagram, such as actual CUDA or OpenCL driver calls).

[0060] 3. Physically heterogeneous computing layer;

[0061] This is the physical entity that ultimately executes the computational task.

[0062] GPU / NPU: Various accelerator hardware. They receive native API instructions from the host machine's "heterogeneous computing power pooling service," perform actual computation and processing, and return the processing results to the virtual machine.

[0063] Therefore, based on the above architecture and the functions of each component, its core data flow (as shown by the arrows in the diagram) is as follows:

[0064] Downlink (request):

[0065] Guest App → (CUDA / OpenCL API) → vAPI driver → (Generate command stream) → VirtIO front-end → (via VirtIO channel) → QEMU back-end → Heterogeneous computing power pooling service → (decoding, scheduling, escaping) → Physical GPU / NPU;

[0066] Upstream (response):

[0067] The calculation results or status information are returned in reverse along the original path and finally delivered to the application inside the virtual machine.

[0068] Of course, in the above architecture and scenarios, the AI ​​application running in the virtual machine can be either a deep learning model training task or an inference service based on a pre-trained model. In addition to running this AI application, it can also run other types of computationally intensive applications. As long as it can be accelerated through standard APIs such as CUDA or OpenCL, it can be included in the scope of this solution.

[0069] In addition to CUDA or OpenCL, the API type can also be other mainstream or emerging heterogeneous computing API standards. As long as it has broad hardware support and good cross-platform characteristics, it can be seamlessly integrated into this solution.

[0070] In addition to QEMU, the virtualization management tools used in the host layer are also compatible with mainstream virtualization platforms such as Xen or VMware; the physical heterogeneous computing power layer supports hardware types that are not limited to GPUs / NPUs, but can also be extended to dedicated accelerators such as FPGAs or ASICs.

[0071] Therefore, based on the above architecture and scenario, the embodiments of this application can achieve efficient interception, escaping, and forwarding of API calls to virtual machines in virtual machine scenarios. Simultaneously, leveraging the lightweight characteristics of the semi-virtualization framework, it significantly reduces the performance overhead of traditional full virtualization solutions. The design of the virtual computing device ensures adaptability to different types of APIs (such as CUDA and OpenCL) and achieves low-latency data transmission between the virtual machine and the host machine through virtualization communication channels (such as standardized VirtIO channels).

[0072] The heterogeneous computing power pooling service at the host layer breaks down the hardware barriers of physical heterogeneous computing power through dynamic scheduling strategies and flexible API escaping logic. This allows resources such as GPUs and NPUs on a single host machine to be shared by multiple virtual machines, significantly improving the utilization of computing resources. Whether it's high-load computing tasks like deep learning model training or latency-sensitive scenarios like real-time inference services, this method can provide near-native hardware acceleration performance for AI applications within virtual machines through optimized command stream processing and hardware scheduling mechanisms, while retaining the strong isolation advantages of virtual machine scenarios and mitigating the security risks associated with shared kernels in containers.

[0073] The following is combined Figure 3 The network framework described herein is used to describe the interface interaction control method in the embodiments of this application:

[0074] Please see Figure 4 One embodiment of the interface interaction control method in this application includes:

[0075] 401. In response to a creation request for a target virtual computing device, create a target virtual computing device in the target virtual environment corresponding to the interface type indicated by the creation request for the target virtual interface;

[0076] The method in this embodiment can be applied to a computer system, which may include a host machine. The computing resources of the host machine can be used to support the operation of a virtual environment. The virtual environment may be an isolated computing environment such as a virtual machine or a container. This embodiment uses a virtual machine as an example for illustration. The target virtual environment is the virtual machine where the target virtual computing power device needs to be deployed. The interface type of the target virtual interface may include CUDA interface, OpenCL interface, etc., corresponding to different heterogeneous computing API standards.

[0077] When a user or management system initiates a request to create a target virtual computing device, the computer system's virtualization management module (such as QEMU) will initialize the corresponding virtual device instance in the target virtual environment according to the interface type specified in the request. For example, if the interface type is CUDA, a virtual computing device that supports vCUDA driver interaction will be created; if it is OpenCL, a virtual computing device that adapts to the vOpenCL driver will be created.

[0078] The virtual interface can be an API, a device file, or other form of interface used to enable communication between the virtual machine and the host machine. When creating the target virtual computing device, ensure that it can emulate the behavior of a real hardware interface so that applications within the virtual machine can interact with it seamlessly. Specifically, this step involves defining the attributes of the target virtual computing device, configuring its communication parameters, and ensuring that it establishes the correct mapping relationship with the actual hardware resources on the host machine (such as GPUs / NPUs). Creating such a target virtual computing device provides efficient processing tools for subsequent command stream transmission and API escaping.

[0079] The target virtual computing device can be created based on the virtual interface between the target virtual environment and the host machine. For example, it can be created based on the type of the virtual interface. If the virtual interface is a CUDA API, the created target virtual computing device will have the ability to hijack, parse, and optimize CUDA API calls, similar to the role of the vCUDA driver mentioned earlier. If the virtual interface is an OpenCL API, the target virtual computing device can process OpenCL API calls accordingly, just like the function of the vOpenCL driver mentioned above.

[0080] 402. The target virtual computing device responds to the deep learning request of the target virtual environment and forwards the deep learning request to the host machine;

[0081] In this step, applications within the target virtual environment (such as AI applications running deep learning tasks) initiate deep learning requests through standard programming interfaces (such as CUDA or OpenCL APIs). These requests are first captured by the target virtual computing device, which has the ability to intercept, parse, and optimize these API calls. After capturing the deep learning requests, the target virtual computing device can perform preliminary processing, such as parsing the parameters in the requests and verifying the validity of the requests. Subsequently, the target virtual computing device forwards the processed deep learning requests to the heterogeneous computing pooling service on the host machine.

[0082] The deep learning request can be a specific heterogeneous computing task instruction, such as a gradient calculation request during model training, a prediction request during model inference, or a matrix operation request during data preprocessing. For example, when an AI application in a virtual machine executes the forward propagation of a convolutional neural network, it calls the convolution calculation function in the cuDNN library through the CUDA API. This call is intercepted by the target virtual computing device (such as the virtual device corresponding to the vCUDA driver). The device parses the key information of the call, such as the input tensor dimension, convolution kernel parameters, stride, and padding. After confirming that the request format is compliant, it encapsulates this information into a standardized command stream and forwards it to the host machine's QEMU backend through the VirtIO channel, awaiting further computing power scheduling and execution.

[0083] 403. Schedule the physical computing power device corresponding to the interface type to respond to the deep learning request, and return the processing result of the deep learning request to the target virtual environment through the target virtual computing power device;

[0084] After the host machine receives a deep learning request from the target virtual computing power device, the heterogeneous computing power pooling service on the host machine initiates the request processing flow. This service first decodes the deep learning request to restore the original API semantics. This step ensures that regardless of the initial API form of the request (such as CUDA or OpenCL), it can be correctly understood and processed. Next, based on the preset computing power allocation strategy and quota, the heterogeneous computing power pooling service determines which physical GPU or NPU will execute the task. This process fully considers factors such as the current load of hardware resources, performance characteristics, and task priority to achieve optimal resource utilization.

[0085] After determining the physical hardware to execute the task, the heterogeneous computing power pooling service further translates the decoded general commands into native API calls corresponding to the target physical hardware. This step is crucial to ensuring that the commands can be executed correctly on the physical hardware. After translation, the native API calls are sent to the corresponding physical GPU or NPU, which performs the actual computation and processing tasks. Once processing is complete, the physical hardware returns the processing results of the deep learning request to the heterogeneous computing power pooling service, which then returns this information to the application within the target virtual environment through the target virtual computing device, thus completing the closed loop of the entire computing task.

[0086] Therefore, this embodiment performs virtualization at the software interaction level, rather than virtualizing the computing hardware. It achieves interaction between different virtual machines and the host machine's physical computing hardware by creating target virtual computing devices. Since the solution is entirely software-based, it does not require the physical GPU to have any special virtualization hardware functions. Therefore, it can manage almost all mainstream and even older GPUs on the market, as well as accelerators such as NPUs with different architectures. Users do not need to pay for expensive virtualization-specific hardware and software licenses, reducing virtualization deployment costs. At the same time, the application targets a stable virtual API interface, rather than a vendor's native driver, thus liberating the application from specific hardware ecosystems, avoiding dependence on a single hardware vendor, and improving the scalability and flexibility of virtualization deployment.

[0087] When it is necessary to upgrade, expand or replace physical computing power equipment, only the hardware adaptation and computing power pool update need to be completed on the host side. The computing services in the target virtual environment can seamlessly call the new physical computing power resources without modifying any code or recompiling. This greatly reduces the operation and maintenance costs and business migration risks of heterogeneous computing power clusters and improves the resource elasticity of computing power hardware.

[0088] In this application embodiment, heterogeneity refers to different types of computing hardware architectures, specifically including but not limited to GPUs (Graphics Processing Units, such as NVIDIA CUDA architecture GPUs and AMD ROCm architecture GPUs), NPUs (Neural Processing Units), and ASICs (Application-Specific Integrated Circuits, such as custom chips designed for specific AI tasks), and other hardware devices with AI computing capabilities. These hardwares differ fundamentally in instruction set architecture, memory management mechanisms, and computing power scheduling logic, making it difficult for traditional virtualization solutions to achieve unified resource pooling and scheduling across architectures. This application embodiment constructs a unified vAPI abstraction layer to shield the technical details of the underlying heterogeneous hardware: on the virtual machine side, applications only need to call standard CUDA or OpenCL APIs without needing to pay attention to the underlying hardware type; on the host machine side, the computing power scheduling service adapts to the driver interfaces and instruction sets of different hardware, converting the virtual machine's API requests into instructions executable by the corresponding hardware, thus achieving transparent virtualization of heterogeneous computing power.

[0089] For example, when an AI application within a virtual machine initiates a CUDA kernel function execution request, if the available resource in the computing power pool is a Huawei Ascend NPU, the computing power scheduling service will automatically translate the CUDA instructions into AscendCL instructions supported by the Ascend NPU. After completing the computation, the result will be returned to the virtual machine application in the form of a CUDA API response. The entire process is completely transparent to the application. This unified virtualization capability of heterogeneous computing power breaks down the ecosystem barriers of hardware manufacturers, allowing users to flexibly combine different types of computing power resources according to business needs, improving resource utilization while reducing the complexity of application development and deployment.

[0090] based on Figure 4 In one optional implementation of the illustrated embodiment, the computer system may further create a transmission channel in the target virtual environment corresponding to the interface type of the target virtual interface indicated by the creation request. Subsequently, the deep learning request can be forwarded to the host machine via this transmission channel, and the processing result of the deep learning request can be transmitted from the host machine to the target virtual computing device via this transmission channel.

[0091] For example, if the target virtual interface is of type CUDA, the system will create a dedicated transmission channel based on VirtIO within the target virtual environment. This channel uses a combination of memory sharing and interrupt notification mechanisms. The vCUDA driver on the virtual machine side can directly write the encapsulated CUDA command stream into the shared memory area without going through the traditional device emulation I / O path, which can significantly reduce transmission latency compared to a fully virtualized solution. If the interface type is OpenCL, an asynchronous transmission channel adapted to the characteristics of OpenCL command queues will be created, supporting batch packaging and sequential execution marking of command streams. When an OpenCL application in the virtual machine initiates multiple kernel function calls, the channel can automatically merge consecutive small requests into large command packets, reducing the number of cross-layer communications and improving transmission efficiency.

[0092] Therefore, by creating a dedicated transmission channel that matches the interface type, the communication link between the virtual machine and the host machine was further optimized, achieving efficient flow of computing power requests and results. The design of this transmission channel fully adapts to the characteristics of different APIs: for example, CUDA's synchronous / asynchronous execution mode and OpenCL's command queue scheduling logic. Through customized channel parameters (such as buffer size, transmission triggering mechanism, and priority queue), it ensures that various computing power requests meet both low-latency requirements and adapt to the hardware's parallel processing capabilities during transmission. This collaborative design of the transmission channel and virtual computing power devices constructs an efficient end-to-end path from virtual machine applications to the host machine's physical computing power, providing effective support for performance improvement in heterogeneous computing power virtualization.

[0093] based on Figure 4In one optional implementation of the illustrated embodiment, the target virtual computing device can optimize the original deep learning request initiated by the virtual machine to the virtual interface, and then forward the optimized deep learning request to the host machine. Specifically, when an original deep learning request initiated by a computing service running in the target virtual environment to the target virtual interface is detected, the target virtual computing device can intercept the original deep learning request to obtain the parameter information of the target virtual interface, and optimize the original deep learning request based on the parameter information of the target virtual interface to obtain the deep learning request. The target virtual computing device can then forward the deep learning request to the host machine and return a successful task submission message to the target virtual environment, so that the computing service can continue to execute the deep learning task based on the successful task submission message.

[0094] The computing service can be an application running in the target virtual environment, such as an AI computing application, a machine learning model training program, or a real-time inference service. The parameter information of the target virtual interface refers to the specific configuration and data details related to the computing task contained in the deep learning request. For example, in CUDA API calls, this includes thread block size, grid dimensions, memory allocation parameters, kernel function names, and parameter passing methods; or in OpenCL API calls, it includes device context, command queue configuration, memory object handles, and kernel program parameter binding information. This parameter information determines the execution logic, resource requirements, and data processing methods of the deep learning request.

[0095] By intercepting raw deep learning requests and extracting their parameter information, the target virtual computing device can perform targeted optimizations based on task characteristics. For example, it can merge repetitive memory copy operations, dynamically adjust kernel function execution parameters to match the host machine's physical hardware computing capabilities, or batch process consecutive API calls to reduce communication overhead. The optimized deep learning requests, while maintaining the original task semantics, can more efficiently utilize the host machine's computing resources, improving overall computing performance.

[0096] Returning the processing result of the original deep learning request to the target virtual environment, that is, returning the information that "the task initiated by the original deep learning request has been successfully submitted to the host machine", aims to ensure that the computing services within the virtual machine can continue to run according to the normal execution logic without waiting for the actual hardware processing of the deep learning request to be completed, thereby realizing asynchronous execution of computing tasks and further reducing the waiting latency of applications within the virtual machine; it also does not need to be aware of the heterogeneity of the underlying hardware and the existence of virtualization processing, and always maintains transparency to the upper-layer applications.

[0097] Specifically, after the target virtual computing device forwards the optimized deep learning request to the host machine, the host machine's heterogeneous computing pooling service will complete command decoding, hardware scheduling, and native API escaping according to the process described above. Finally, the physical hardware will execute the computation and return the results. These results will be encapsulated by the heterogeneous computing pooling service into a response format consistent with the semantics of the original deep learning request, and then sent back to the computing service through the target virtual computing device. Upon receiving the response, the computing service will treat it as a result of direct interaction with the physical hardware and continue executing subsequent task steps.

[0098] For example, in deep learning training, model parameters are updated based on gradients; in real-time inference services, business responses are generated based on output results. This transparent result return mechanism eliminates the need for special adaptations to virtualization environments in the development and deployment of computing services, ensuring full compatibility with native API calling logic. This reduces the cost of application migration while guaranteeing the continuity and stability of business processes.

[0099] When optimizing the original deep learning request, the parameter information of the target virtual interface can be parsed to determine the processing type of the original deep learning request. Then, the deep learning request of the target virtual interface can be optimized according to the processing type of the original deep learning request to obtain the deep learning request.

[0100] This processing type can include various types such as information query type, asynchronous task request type, and resource synchronous request type. The processing type of the original deep learning request is determined based on the parameter information of the virtual interface. For information query type requests, this can be determined by analyzing whether the parameters contain key fields such as "query" or "get" (e.g., parameters in CUDA's cuDeviceGetAttribute or OpenCL's clGetDeviceInfo calls), or by determining whether the request only needs to return hardware status and resource information without triggering an actual computation task.

[0101] For asynchronous task request types, it can be determined whether the "asynchronous" flag exists in the parameters (such as the asynchronous execution flag specified in cudaLaunchKernelAsync of CUDA or clEnqueueNDRangeKernel of OpenCL), or whether the request allows the computation task to be executed in the background without needing to return results immediately.

[0102] The resource synchronization request type refers to the resource call request of the deep learning task in the target virtual environment, such as the call request for memory resources, data resources, etc. Specifically, it can be determined by judging whether the parameter information of the target virtual interface indicates a request for resources and the quantity or scale of resources to be acquired. For example, requesting 2GB of video memory or copying 1KB of data indicates a resource acquisition request. Based on this, it can be determined that the deep learning request belongs to the resource synchronization request type.

[0103] Specifically, when optimizing deep learning requests, if the original deep learning request is an information query type, it is restructured as a query request for information in a pre-defined cache space. Then, when returning the processing result of the original deep learning request to the target virtual environment, the target information queried by the target virtual interface can be determined in this cache space and the queried target information can be returned to the target virtual environment.

[0104] For example, the original deep learning request could be a query for static information such as device model and version (e.g., GPU model, memory size, etc.). Since this type of information typically doesn't change after the virtual machine starts, the target virtual computing device can pre-obtain and cache this information from the host machine's heterogeneous computing pooling service during the initialization phase. When an application within the virtual machine initiates an information query request, the target virtual computing device directly retrieves the corresponding static data, such as device model, driver version, and number of computing cores, from its local cache, without forwarding the request to the host machine's physical hardware. This optimization transforms query requests that originally required cross-layer communication into local cache reads, reducing response latency to the microsecond level while avoiding invalid access to physical hardware and reducing the processing load on the host machine.

[0105] If the processing type of the original deep learning request is an asynchronous task request type, then the original deep learning request can be constructed as a deep learning request for a subtask of the deep learning task running in the target virtual environment. Here, the subtask refers to one or more subdivided tasks of the deep learning task running in the virtual environment. For example, when the virtual environment runs a deep learning task, it initiates a deep learning request to request the execution of a kernel function; that is, "executing the kernel function" is a subtask of the deep learning task running in the virtual environment. A deep learning request for this subtask can be constructed and sent to the host machine.

[0106] For example, the target virtual computing device intercepts API calls (i.e., the original deep learning requests) initiated by a deep learning model running in the virtual environment. Parsing the API call reveals its semantics as "execute kernel function," and the device then constructs a "execute kernel function" deep learning request. This "execute kernel function" deep learning request can be sent to the host machine, where the host's heterogeneous computing scheduling service allocates the kernel function task to the appropriate computing hardware for processing based on the current resource load of the computing pool. Simultaneously, the target virtual computing device immediately returns an asynchronous response to the deep learning model within the virtual machine: "Kernel function task successfully submitted." The model can continue executing subsequent task scheduling logic, such as preparing the next batch of training data or initiating new kernel function calls, without waiting for the hardware to complete. This asynchronous processing mechanism allows applications within the virtual machine to fully utilize time slices, avoiding execution blocking caused by hardware waiting and significantly improving the overall parallelism of the tasks.

[0107] Therefore, when returning the processing result of the original deep learning request to the target virtual environment, a task submission success message can be returned to the target virtual environment. This message indicates that the task initiated by the computing service has been successfully submitted to the host machine's computing power module, allowing the computing service to perform other operations while waiting for the actual task processing result, rather than simply waiting for it. Furthermore, the task processing request can be sent to the host machine's computing power module, which then processes the task initiated by the computing service.

[0108] For example, the original deep learning request of this asynchronous request type could be a kernel function execution request, such as a kernel function in GPU / parallel computing. These requests typically require high computational accuracy but have low dependence on immediate response. When the target virtual computing device detects this type of asynchronous request, it can first parse the kernel function parameters in the request, including key information such as thread block dimension, shared memory size, and memory addresses of input and output data. The target virtual computing device can then generate a deep learning request containing the above parameters and forward it to the host machine's heterogeneous computing pooling service. Simultaneously, the target virtual computing device immediately returns a "task submission successful" status code (such as cudaSuccess for CUDA or CL_SUCCESS for OpenCL) to the computing service within the virtual machine, allowing the computing service to continue executing subsequent task scheduling or data preparation steps without waiting for the actual computation to complete.

[0109] Upon receiving a task processing request, the host machine's computing module can allocate the task to idle computing cores based on the current GPU / NPU load and trigger asynchronous execution of kernel functions via native API calls. Once computation is complete, the computing module writes the result to a pre-defined shared memory region and pushes a "task complete" notification to the computing service in the virtual environment through the target virtual computing device. The computing service then reads the final result from the shared memory, achieving an efficient closed loop for asynchronous tasks.

[0110] If the processing type of the original deep learning request is a resource synchronization request, then multiple original deep learning requests corresponding to the resource synchronization request type can be merged into a single resource request command to obtain the deep learning request. Furthermore, when returning the processing result of the original deep learning request to the target virtual environment, the virtual resource information corresponding to the multiple original deep learning requests can be returned to the target virtual environment. Moreover, the deep learning request corresponding to this single resource request command can be sent to the host machine's processing module, which can then process the single resource request command.

[0111] The virtual resource information can be the virtual address of the resource, the allocated virtual storage capacity, or the logical identifier of the resource, and is completely decoupled from the actual resource address or allocation method of the physical hardware. For example, when the target virtual computing power device detects that the computing service in the target virtual environment initiates three consecutive CUDA API memory allocation requests in a short period of time (such as calling cudaMalloc to request 512MB, 1GB, and 768MB of memory respectively), and these requests are all of the resource synchronization request type, the target virtual computing power device can merge and optimize these requests, and return a continuous virtual memory address space to the target virtual environment (for example, a continuous virtual address segment from 0x10000000 to 0x1C000000, with a total capacity of 2.25GB), while merging these three scattered requests into a physical memory request with a total capacity of 2.25GB and forwarding it to the host machine's heterogeneous computing power pooling service.

[0112] After receiving the merged request, the host machine can allocate a contiguous physical memory region to the virtual machine based on the remaining physical GPU memory, and map the physical address of this region to the virtual address space returned by the target virtual computing device. When the computing service subsequently reads or writes data through the virtual address, the target virtual computing device will automatically complete the conversion from virtual address to physical address to ensure the correctness of data access.

[0113] Furthermore, deep learning requests of the resource synchronization request type can also be requests for memory copies, which refer to copying data between the CPU's system memory and the GPU's video memory. The reason for the need for memory copies is that the GPU's computing cores cannot directly access the CPU's system memory, while the input data required for computing tasks is usually stored in system memory, and the computing results also need to be transferred back from video memory to system memory for subsequent processing.

[0114] Therefore, when multiple deep learning requests are memory copy requests, the target virtual computing device can perform batch processing optimization on these requests, merging multiple scattered memory copy operations into a single continuous data transfer. For example, if the computing service within the target virtual environment initiates four cudaMemcpy calls within 100 milliseconds, copying 128MB, 256MB, 64MB, and 192MB of data from system memory to GPU memory respectively, the target virtual computing device can intercept these requests and analyze their data flow and the continuity between source and target addresses.

[0115] If the source data for these copy requests is detected to be stored contiguously in system memory (e.g., a contiguous address range from 0x20000000 to 0x26000000, totaling 640MB), and the target video memory address can also be merged into a contiguous region (e.g., from 0x30000000 to 0x36000000), the target virtual computing device can merge these four independent requests into a single batch copy request with a total capacity of 640MB, and generate corresponding virtual resource information (e.g., a merged virtual source address and target address mapping table) and return it to the computing service. This allows the computing service to perceive that the four copy operations have been "successfully completed" and continue executing subsequent logic.

[0116] Simultaneously, the target virtual computing device forwards the merged batch requests to the host machine's heterogeneous computing pooling service. This service initiates a continuous data transfer via native APIs (such as CUDA's cudaMemcpyAsync or the NPU's proprietary memory copy interface). This merging optimization not only reduces the communication overhead of API calls (from four cross-layer interactions to one) but also leverages the hardware's DMA (Direct Memory Access) continuous transfer characteristics, significantly improving data transfer efficiency, especially when handling small-batch, high-frequency copy requests, thus avoiding performance degradation caused by frequent hardware context switching.

[0117] In addition, the target virtual computing power device can monitor the merged transmission process. After the physical hardware completes the data copy, the target virtual computing power device will synchronize the actual transmission status to the virtual resource management module of the computing service, ensuring the consistency of the status of virtual resources and physical resources. This satisfies the data integrity requirements of resource synchronization requests and achieves efficient utilization of computing resources through batch processing.

[0118] In some alternative implementations, when intercepting the deep learning request, the deep learning request can be intercepted based on the driver module of the target virtual computing device; or, a preset API hijacking tool can be loaded, and the deep learning request can be intercepted based on the API hijacking tool.

[0119] The API hijacking tool refers to the tool that can intercept deep learning requests from target processes to specific computing APIs (such as CUDA, OpenCL, ROCm, etc.) at the operating system level. By modifying the import address table (IAT) or function pointers of dynamic link libraries (DLLs) of the process, it redirects the original API calls to the processing logic of the target virtual computing device.

[0120] For example, this API hijacking tool could be an API hijacking library that injects itself into the process space of a computing service during the operating system startup phase of the target virtual environment, modifying the entry addresses of functions related to the computing API within that process. When the computing service calls native APIs such as cudaMalloc, clEnqueueNDRangeKernel, the request is automatically redirected to a hook function in the API hijacking library. The hook function then extracts the call parameters and forwards them to the target virtual computing device for further processing.

[0121] In Windows systems, API hijacking tools can also inject DLLs into the compute service process space within a virtual machine, replacing the entry addresses of functions such as cuLaunchKernel and cuMemAlloc in CUDA-related DLLs (such as cudart64_110.dll) within the process. In Linux systems, the LD_PRELOAD environment variable can be used to force the process to preferentially load shared libraries containing hijacking logic (such as libcuda_hook.so), thereby capturing all calls to functions in libcuda.so.

[0122] This hijacking method does not require modification of the virtual machine kernel or hardware drivers. It can capture computing API calls through dynamic interception in user space, and is characterized by flexible deployment and low intrusion into the virtual machine system.

[0123] The driver module of the target virtual computing device can intercept the raw deep learning request by registering driver callback functions related to the computation API interaction in the virtual machine kernel. When the computing service initiates a deep learning request for the physical heterogeneous computing power, the request will first trigger the callback logic of the driver module. For example, in a Linux system, the driver module of the target virtual computing device can implement the file_operations structure of the character device driver to bind system calls such as read, write, and ioctl with the processing logic of the virtual interface. When the computing service accesses the target virtual computing device file (such as / dev / virtual_hw) through a system call, the driver module will intercept the system call, extract the API call parameters (such as the called function name, input / output buffer address, parameter length, etc.), and encapsulate these parameters into a standardized raw deep learning request structure.

[0124] This driver-module-based interception method operates directly at the operating system kernel layer, enabling more efficient and reliable capture of all requests interacting with physical computing power. It avoids issues such as process injection failure and hook function overriding that may occur with user-space API hijacking, making it particularly suitable for virtualization scenarios with high security and stability requirements.

[0125] based on Figure 4 In the illustrated embodiments and their various optional implementations described above, when creating a target virtual computing power device corresponding to the interface type of the target virtual interface, one optional implementation is to create a computing power pool based on multiple physical computing power hardware, and, according to the interface type of the target virtual interface indicated by the creation request, create a target virtual computing power device corresponding to that interface type in the target virtual environment. Furthermore, determine preset computing power resources in the computing power pool to be associated with the target virtual computing power device, and attach the target virtual computing power device to the target virtual environment. Therefore, when the host machine processes a deep learning request and returns a processing result, the host machine can schedule the physical computing power device corresponding to that computing power resource to respond to the deep learning request and return the processing result of the deep learning request to the target virtual environment through the target virtual computing power device.

[0126] For example, the host machine can pre-abstract and pool resources for different types of physical computing hardware such as GPUs, NPUs, and FPGAs within the physical heterogeneous computing power layer. When creating a target virtual computing power device, a preset number or specification of computing power resources can be selected from the computing power pool and bound to the target virtual computing power device according to the business needs of the target virtual environment. For example, if an AI training scenario requires high FP16 computing power, the target virtual computing power device can be associated with a GPU computing power node; if a real-time data processing scenario requires low latency, the target virtual computing power device can be associated with an FPGA computing power node.

[0127] For example, to create a target virtual environment for running deep learning training tasks, two GPUs from the computing power pool (let's assume resource identifiers GPU-003 and GPU-004) can be bound together. The total video memory (80GB) and computing cores (108 SM units) of these two GPUs can be abstracted into the "virtual computing power specifications" of the target virtual computing power device. Finally, through the device pass-through mechanism of the hypervisor (such as QEMU-KVM), the target virtual computing power device is connected to the PCIe bus of the target virtual environment, allowing the computing services within the virtual machine to recognize the target virtual computing power device through standard computing APIs (such as CUDA) and call it as "local physical computing hardware".

[0128] When a computing service within the target virtual environment initiates a kernel function execution request, the target virtual computing device intercepts the request and sends a resource request to the host machine's computing scheduling service based on the bound computing resource identifier. The scheduling service queries the load of GPU-003 and GPU-004 in real time (e.g., GPU-003 is currently at 30% load, and GPU-004 is at 25% load), and allocates the request to the GPU-004 with the lower load. After GPU-004 completes the computation, it sends the result back to the computing service of the target virtual environment through the target virtual computing device, achieving transparent access to physical computing resources.

[0129] Therefore, by creating target virtual computing devices and associating them with computing resources, unified scheduling and elastic allocation of various types of hardware resources within the physical heterogeneous computing layer can be achieved, avoiding resource idleness caused by a single virtual machine monopolizing a certain type of computing hardware. For example, when a GPU becomes saturated due to handling high-load tasks, the computing scheduling service can automatically offload subsequent tasks to idle GPUs of the same type or NPUs with compatible computing power. The target virtual computing device then dynamically adjusts the resource mapping relationship to ensure that the computing services within the virtual machine can adapt to the computing characteristics of different physical hardware without modifying the code.

[0130] This resource pooling and dynamic binding mechanism not only improves the resource utilization of the physical heterogeneous computing layer, but also provides virtual machines with flexible computing power expansion capabilities. It can adjust the specifications of the bound computing resources in real time according to changes in business load, meeting diverse needs from low-load lightweight computing to high-load parallel computing.

[0131] In this embodiment of the application, the target virtual computing power device can be a vAPI device, and the architecture of the vAPI device can be illustrated as follows. Figure 5 As shown, it describes the core system architecture and the deployment locations and relationships of key components of the heterogeneous computing power virtualization scheme in this application embodiment. The architecture is explained in detail below:

[0132] 1. Virtual machine side components (running in the Guest OS);

[0133] This part is responsible for intercepting and processing application requests inside the virtual machine.

[0134] API hijacking library (optional): A dynamic library running in user space. Its main responsibility is to capture raw CUDA / OpenCL API calls issued by applications (i.e., computing services running on virtual machines) through library link hijacking (such as LD_PRELOAD) and perform preliminary parsing. Marked as "optional," this functionality can also be integrated into the underlying driver.

[0135] vAPI device driver: A core component running in the virtual machine kernel space. It drives the virtual PCIe device (i.e., the vAPI device frontend) emulated by Qemu. Its core responsibility is to receive API calls from API hijacking libraries or directly intercepted API calls, perform deep parsing, semantic understanding and optimization (such as asynchronous returns and request merging), convert the optimized requests into an internal command stream, and pass it to the vAPI device frontend through the driver interface to be sent to the next node.

[0136] 2. Virtualization layer and interface components (the bridge connecting the Guest and the Host).

[0137] This part is responsible for establishing a reliable and efficient data channel between the virtual machine and the host machine.

[0138] vAPI device front end: A standard VirtIO PCIe device simulated by Qemu. It appears as a hardware device within the virtual machine, and its function is to receive command streams from the driver and send them to the corresponding back end of the host machine through semi-virtualized I / O mechanisms such as VirtIO.

[0139] Qemu: An open-source machine emulator and virtualizer, it is the execution engine for device emulation and I / O virtualization, responsible for implementing the emulation behavior of the "vAPI device front end" and running the "vAPI device back end".

[0140] vAPI device backend: A component running within the Qemu process, paired with the frontend. It is responsible for receiving data packets from the vAPI device frontend through the virtualization channel, unpacking and performing preliminary processing, and then delivering the parsed requests to the core services on the host machine.

[0141] 3. Host-side components (running in the Host OS);

[0142] Computing scheduling service: A core backend service on the host machine. It receives requests from multiple virtual machines and multiple vAPI device backends, playing several key roles: as a resource scheduler, it determines which specific physical GPU / NPU to assign the current computing task to based on the configuration of the vAPI device (quota, to which computing pool it belongs); as a protocol translator, it translates the general internal command stream into the native APIs required by the target physical hardware (such as actual CUDA driver calls) and executes them.

[0143] vAPI device management and heterogeneous computing device management are control and management plane components of the system, typically existing in the form of management services or APIs. Among them:

[0144] Heterogeneous computing device management is responsible for the management of physical hardware, including discovering physical GPUs / NPUs, monitoring their status, and grouping them into different computing pools.

[0145] vAPI device management is responsible for the lifecycle management of virtual devices. Based on administrator instructions, it uses Qemu commands to create / delete vAPI front-end devices and associates them with configurations (such as quotas and bound computing pools) in the back-end and computing power scheduling service.

[0146] Therefore, based on this architecture, a clear hierarchical division of labor can be achieved, namely, a clear division between the request processing and optimization layer on the client side (within the Guest), the transport layer (VirtIO), and the resource scheduling and execution layer on the server side (Host), each performing its own function with low coupling. Simultaneously, the control plane and data plane are separated. For example, the data plane forms an efficient data processing and forwarding pipeline based on the link of "API hijacking → vAPI driver → front-end → back-end → computing power scheduling service," while the control plane, based on vAPI device management and heterogeneous computing power device management, is responsible for the static configuration of resources and policy distribution, without intervening in the real-time data stream, thus ensuring low latency and high throughput of data transmission.

[0147] Figure 6 This demonstrates the configuration process of creating resource pools, allocating virtual devices, and distributing configurations to the entire system through a management interface or API. Specifically, when an administrator executes the "Create Pool" operation through the management interface or API in the Web console, the command reaches the "Heterogeneous Computing Power Device Management" service. This service is responsible for logically organizing the specified physical GPU / NPU hardware resources into an allocable computing power resource pool and recording the configuration.

[0148] Next, the administrator executes the "Create Device and Assign Device" operation. This command is handled by the "VAPI Device Management" service, whose core responsibility is the lifecycle management of virtual devices. This service creates a virtual device record in the database based on the administrator's selected computing pool, specified API type (such as vCUDA), and resource configuration (such as video memory size), and binds it to the target virtual environment. Subsequently, to actually represent this device on the virtual machine, "VAPI Device Management" issues a "Command to Add Settings Simulation" command to the "Qemu" process. Upon receiving the command, Qemu dynamically simulates the corresponding vAPI device front-end (i.e., the virtual PCIe device) when the target virtual environment starts up or is hot-plugged.

[0149] Finally, to ensure that virtual devices can use resources correctly, critical quotas and scheduling policies need to be synchronized. Therefore, "VAPI Device Management" sends specifications to the "Compute Scheduling Service." After receiving the virtual device's ID, its associated compute pool, and specific compute and memory quotas, the Compute Scheduling Service updates its internal scheduling table. At this point, the entire process of a target virtual compute device, from logical configuration to actual usability, is complete. The driver within the virtual machine can begin sending and receiving requests through this target virtual compute device, and the Compute Scheduling Service is ready to allocate resource quotas and route tasks according to the sent specifications.

[0150] Therefore, the illustrated backend management logic encapsulates the complex physical resource virtualization process into a simple two-step operation: "create pool" and "create device." Through the reliable linkage of a series of backend services, management intentions are seamlessly transmitted to various runtime components, ensuring that administrators can quickly complete the virtualization configuration of heterogeneous computing resources without needing in-depth understanding of the underlying technical details. This design significantly lowers the operational threshold, enabling administrators with different technical backgrounds to efficiently manage resource pooling and deploy virtual devices. It also provides fundamental support for the automated operation and maintenance of large-scale heterogeneous computing clusters.

[0151] Figure 7 This illustrates the process by which the vAPI device driver handles three different types of API calls initiated by AI applications within the virtual machine and interacts with the backend via the Virtio channel. The core process is explained in detail below:

[0152] Part 1: Request Initiation and Interception;

[0153] Guest ML (machine learning / AI applications within a virtual machine) issues three types of requests:

[0154] Model-Configuration Class: Static requests that query device attributes and version;

[0155] Asynchronous requests: such as the kernel launch function (cudaLaunchKernel), which does not require waiting for the result immediately after being called;

[0156] Synchronization requests, such as allocating video memory (cudaMalloc) and synchronous memory copying, require waiting for the operation to complete after being called.

[0157] Part Two: Intelligent Processing Driven by vAPI;

[0158] Once a request is intercepted by the vAPI driver (vCuda / vOpenCL), it is handled differently based on its type, which is a manifestation of performance optimization:

[0159] For "model-configuration" requests, the vAPI driver directly reads the pre-configured virtual device information (such as the virtual GPU model and video memory size) from its internal cache and immediately returns it to the application in the virtual machine.

[0160] For "asynchronous requests," the vAPI driver immediately returns a "success" signal (such as cudaSuccess) to the application, allowing the application to continue executing subsequent code without waiting. Simultaneously, the request is placed in an internal queue, potentially merged with other requests, and then asynchronously forwarded to the host machine's actual hardware for execution via the Virtio channel.

[0161] For "synchronization requests", the vAPI driver can try to merge multiple consecutive synchronization requests (such as multiple memory copies) into a single batch request before forwarding them to reduce the number of communications.

[0162] Part Three: Transmission and Backend

[0163] Channel: All requests that need to be forwarded to the host machine are transmitted efficiently through the Virtio device channel (implemented by vAPI device front-end + back-end).

[0164] On the host machine side: The request is received by the computing power scheduling service, which performs resource scheduling and API translation, and finally executes it on the physical GPU / NPU. The result is then returned along the original path.

[0165] Therefore, this process not only achieves refined processing of different types of API requests, but also effectively reduces the communication overhead between the virtual machine and the host machine through optimization strategies such as asynchronous forwarding and batch merging, thereby improving the overall efficiency of heterogeneous computing power calls.

[0166] The computer system described in the embodiments of this application is described below. One embodiment of the computer system in this application includes:

[0167] The computer system includes a host machine, whose computing resources are used to support the operation of the virtual environment; the computer system includes:

[0168] A creation unit is configured to respond to a creation request for a target virtual computing power device and, based on the interface type of the target virtual interface indicated by the creation request, create a target virtual computing power device in the target virtual environment corresponding to the interface type.

[0169] The sending unit is configured to respond to the deep learning request of the target virtual environment through the target virtual computing power device, and forward the deep learning request to the host machine;

[0170] The scheduling unit is used to schedule the physical computing power device corresponding to the interface type to respond to the deep learning request, and return the processing result of the deep learning request to the target virtual environment through the target virtual computing power device.

[0171] In a preferred embodiment of this example, the creation unit is further configured to:

[0172] Create a transmission channel in the target virtual environment that corresponds to the interface type;

[0173] The deep learning request is forwarded to the host machine via the transmission channel, and the processing result of the deep learning request is transmitted from the host machine to the target virtual computing device via the transmission channel.

[0174] In a preferred embodiment of this invention, the creation unit is specifically used for:

[0175] A computing power pool is created based on multiple physical computing power hardware, and a target virtual computing power device corresponding to the interface type is created in the target virtual environment according to the interface type, and a preset computing power resource is determined in the computing power pool and associated with the target virtual computing power device.

[0176] Connect the target virtual computing device to the target virtual environment;

[0177] The scheduling unit is specifically used for:

[0178] The physical computing devices corresponding to the computing resources are scheduled to respond to the deep learning request, and the processing results of the deep learning request are returned to the target virtual environment through the target virtual computing device.

[0179] In a preferred embodiment of this invention, the transmitting unit is specifically used for:

[0180] When a raw deep learning request initiated by a computing service running in the target virtual environment to the target virtual interface is detected, the raw deep learning request is intercepted based on the target virtual computing power device and the parameter information of the target virtual interface is obtained.

[0181] The target virtual computing power device optimizes the original deep learning request based on the parameter information of the target virtual interface to obtain the deep learning request;

[0182] The target virtual computing device forwards the deep learning request to the host machine and returns a successful task submission message to the target virtual environment, so that the computing service can continue to execute the deep learning task based on the successful task submission message.

[0183] In a preferred embodiment of this invention, the transmitting unit is specifically used for:

[0184] Parse the parameter information of the target virtual interface to determine the processing type of the original deep learning request;

[0185] The original deep learning request is optimized according to the processing type of the original deep learning request to obtain the deep learning request.

[0186] In a preferred embodiment of this invention, the transmitting unit is specifically used for:

[0187] If the processing type of the original deep learning request is an asynchronous task request type, then the original deep learning request is constructed as a deep learning request for a subtask of the deep learning task running in the target virtual environment.

[0188] If the processing type of the original deep learning request is a resource synchronization request type, then the multiple original deep learning requests corresponding to the resource synchronization request type are merged into a single resource request command to obtain the deep learning request.

[0189] In a preferred embodiment of this example, if the processing type of the original deep learning request is an information query type, the sending unit is further configured to:

[0190] In response to the original deep learning request's query request for information in the preset cache space, the target information queried by the target virtual interface is determined in the cache space, and the target information is returned to the target virtual environment.

[0191] In a preferred embodiment of this invention, the computer system further includes an acquisition unit, used for:

[0192] The deep learning request is intercepted by the driver module of the target virtual computing device; or, a preset API hijacking tool is loaded, and the deep learning request is intercepted based on the API hijacking tool.

[0193] In this embodiment, the operations performed by each unit in the computer system are the same as those described above. Figure 4 The embodiments shown and their various alternative implementations are similar to those described herein, and will not be repeated here.

[0194] The computer device in the embodiments of this application is described below. Please refer to [link / reference]. Figure 8 One embodiment of the computer device in this application includes:

[0195] The computer device 800 may include one or more central processing units (CPUs) 801 and a memory 805, in which one or more applications or data are stored.

[0196] The memory 805 can be volatile or persistent storage. The program stored in the memory 805 can include one or more modules, each module including a series of instruction operations on the computer device. Furthermore, the central processing unit 801 can be configured to communicate with the memory 805 and execute the series of instruction operations in the memory 805 on the computer device 800.

[0197] The computer device 800 may also include one or more power supplies 802, one or more wired or wireless network interfaces 803, one or more input / output interfaces 804, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0198] The central processing unit 801 can perform the aforementioned... Figure 4 The operations performed by the computer device in the illustrated embodiments and their various alternative embodiments are not described in detail here.

[0199] This application also provides a computer storage medium, one embodiment of which includes: the computer storage medium storing instructions, which, when executed on a computer, cause the computer to perform the aforementioned... Figure 4 The operations performed by the computer device in the illustrated embodiments and their various alternative embodiments.

[0200] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0201] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0202] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0203] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0204] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An interface interaction control method, characterized in that, The method is applied to a computer system, the computer system including a host machine, the computing resources of which are used to support the operation of a virtual environment; the method includes: In response to a request to create a target virtual computing device, a target virtual computing device corresponding to the interface type of the target virtual interface indicated in the creation request is created in the target virtual environment. The target virtual computing device responds to the deep learning request of the target virtual environment and forwards the deep learning request to the host machine; The system schedules the physical computing device corresponding to the interface type to respond to the deep learning request, and returns the processing result of the deep learning request to the target virtual environment through the target virtual computing device.

2. The method according to claim 1, characterized in that, The method further includes: Create a transmission channel in the target virtual environment that corresponds to the interface type; The deep learning request is forwarded to the host machine via the transmission channel, and the processing result of the deep learning request is transmitted from the host machine to the target virtual computing device via the transmission channel.

3. The method according to claim 1, characterized in that, The step of responding to the deep learning request of the target virtual environment through the target virtual computing power device and forwarding the deep learning request to the host machine includes: When a raw deep learning request initiated by a computing service running in the target virtual environment to the target virtual interface is detected, the raw deep learning request is intercepted based on the target virtual computing power device and the parameter information of the target virtual interface is obtained. The target virtual computing power device optimizes the original deep learning request based on the parameter information of the target virtual interface to obtain the deep learning request; The target virtual computing device forwards the deep learning request to the host machine and returns a successful task submission message to the target virtual environment, so that the computing service can continue to execute the deep learning task based on the successful task submission message.

4. The method according to claim 3, characterized in that, The step of optimizing the original deep learning request based on the parameter information of the target virtual interface to obtain the deep learning request includes: Parse the parameter information of the target virtual interface to determine the processing type of the original deep learning request; The original deep learning request is optimized according to the processing type of the original deep learning request to obtain the deep learning request.

5. The method according to claim 4, characterized in that, The step of optimizing the original deep learning request according to the processing type of the original deep learning request to obtain the deep learning request includes: If the processing type of the original deep learning request is an asynchronous task request type, then the original deep learning request is constructed as a deep learning request for a subtask of the deep learning task running in the target virtual environment. If the processing type of the original deep learning request is a resource synchronization request type, then the multiple original deep learning requests corresponding to the resource synchronization request type are merged into a single resource request command to obtain the deep learning request.

6. The method according to claim 3, characterized in that, If the processing type of the original deep learning request is an information query type, then the method further includes: In response to the original deep learning request's query request for information in the preset cache space, the target information queried by the target virtual interface is determined in the cache space, and the target information is returned to the target virtual environment.

7. The method according to claim 1, characterized in that, Obtaining the deep learning request includes: The deep learning request is intercepted by the driver module of the target virtual computing device; or, a preset API hijacking tool is loaded, and the deep learning request is intercepted based on the API hijacking tool.

8. The method according to any one of claims 1 to 7, characterized in that, The step of creating a target virtual computing power device corresponding to the interface type of the target virtual interface indicated by the creation request in the target virtual environment includes: A computing power pool is created based on multiple physical computing power hardware, and a target virtual computing power device corresponding to the interface type is created in the target virtual environment according to the interface type, and a preset computing power resource is determined in the computing power pool and associated with the target virtual computing power device. Connect the target virtual computing device to the target virtual environment; The scheduling of the physical computing power device corresponding to the interface type to respond to the deep learning request, and the return of the processing result of the deep learning request to the target virtual environment through the target virtual computing power device, includes: The physical computing devices corresponding to the computing resources are scheduled to respond to the deep learning request, and the processing results of the deep learning request are returned to the target virtual environment through the target virtual computing device.

9. A computer system, characterized in that, The computer system includes a host machine, whose computing resources are used to support the operation of the virtual environment; the computer system includes: A creation unit is configured to respond to a creation request for a target virtual computing power device and, based on the interface type of the target virtual interface indicated by the creation request, create a target virtual computing power device in the target virtual environment corresponding to the interface type. The sending unit is configured to respond to the deep learning request of the target virtual environment through the target virtual computing power device, and forward the deep learning request to the host machine; The scheduling unit is used to schedule the physical computing power device corresponding to the interface type to respond to the deep learning request, and return the processing result of the deep learning request to the target virtual environment through the target virtual computing power device.

10. The computer system according to claim 9, characterized in that, The creation unit is also used for: Create a transmission channel in the target virtual environment that corresponds to the interface type; The deep learning request is forwarded to the host machine via the transmission channel, and the processing result of the deep learning request is transmitted from the host machine to the target virtual computing device via the transmission channel.

11. The computer system according to any one of claims 9 to 10, characterized in that, The creation unit is specifically used for: A computing power pool is created based on multiple physical computing power hardware, and a target virtual computing power device corresponding to the interface type is created in the target virtual environment according to the interface type, and a preset computing power resource is determined in the computing power pool and associated with the target virtual computing power device. Connect the target virtual computing device to the target virtual environment; The scheduling unit is specifically used for: The physical computing devices corresponding to the computing resources are scheduled to respond to the deep learning request, and the processing results of the deep learning request are returned to the target virtual environment through the target virtual computing device.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

13. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on the computer, cause the computer to perform the method as described in any one of claims 1 to 8.

Images