Methods, apparatus, equipment and storage media for sharing dedicated processing resources

Abstract

This disclosure relates to methods, apparatus, devices, and storage media for sharing dedicated processing resources. The method includes: in response to receiving a request to invoke an interface of the dedicated processing resource, retrieving information associated with the request in the user space of a virtual machine; providing the information to a host running the virtual machine in the kernel space of the virtual machine; and invoking the interface of the dedicated processing resource on the host based on the information.

Description

Technical Field

[0001] Embodiments of this disclosure generally relate to the virtualization of dedicated processing resources, particularly graphics processing unit (GPU) resources, and more specifically to methods, apparatus, devices, and computer-readable storage media for sharing GPU resources. Background Technology

[0002] With the rapid development of artificial intelligence, technologies such as autonomous driving and facial recognition have gradually become familiar to people. The powerful computing power behind these technologies can be provided by a central processing unit (CPU) or a GPU. A CPU consists of a few cores optimized for sequential processing, while a GPU has a large number of smaller, more efficient cores forming a massively parallel computing architecture. GPUs are designed to handle multiple tasks simultaneously and are better suited for parallel computing, thus providing powerful computing power for artificial intelligence.

[0003] Cloud computing platforms, through GPU virtualization technology, can provide users with GPU instances for services such as model training, prediction, and high-performance computing. Currently, the mainstream solution used by cloud platforms is pass-through technology, where a GPU card can only be allocated to one virtual machine instance exclusively; other virtual machine instances cannot access this card. However, in some use cases, such as classroom teaching, each student can use a separate virtual machine. In such scenarios, the required computing power is not very high, and directly using pass-through technology would undoubtedly lead to resource waste. Summary of the Invention

[0004] According to embodiments of this disclosure, a shared access scheme for GPU resources is provided.

[0005] In a first aspect of this disclosure, a method for sharing a dedicated processing resource is provided. The method includes: in response to receiving a request to invoke an interface of the dedicated processing resource, retrieving information associated with the request in user mode of a virtual machine; providing the information to a host running the virtual machine in kernel mode of the virtual machine; and invoking the interface of the dedicated processing resource on the host based on the information.

[0006] In a second aspect of this disclosure, an apparatus for sharing a dedicated processing resource is provided, comprising: an extraction module configured to extract information associated with the request in user mode of a virtual machine in response to receiving a request for invoking an interface of the dedicated processing resource; a providing module configured to provide the information to a host running the virtual machine in kernel mode of the virtual machine; and an invoking module configured to invoke the interface of the dedicated processing resource on the host based on the information.

[0007] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: one or more processors; and a memory for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the method according to a first aspect of this disclosure.

[0008] In a fourth aspect of this disclosure, a computer-readable medium is provided having a computer program stored thereon that, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0009] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram of an exemplary environment in which embodiments of the present disclosure can be implemented is shown;

[0012] Figure 2 A flowchart of a method for sharing GPU resources according to some embodiments of the present disclosure is shown;

[0013] Figure 3 A block diagram of an apparatus for sharing GPU resources according to some embodiments of the present disclosure is shown; and

[0014] Figure 4 A block diagram of an electronic device capable of implementing some embodiments of the present disclosure is shown. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] As mentioned above, pass-through technology is currently commonly used for sharing GPU resources. However, pass-through technology can lead to resource waste in scenarios with relatively low resource consumption. Therefore, embodiments of this disclosure provide a scheme for sharing GPU resources to reduce waste of GPU resources. The following, in conjunction with... Figures 1-3 The embodiments of this disclosure will be described in detail below. It should be understood that although the embodiments of this disclosure are described in conjunction with GPUs, the embodiments of this disclosure can also be applied to any other suitable dedicated processing resources.

[0018] Figure 1 A schematic diagram of an exemplary computing environment 100 in which embodiments of the present disclosure can be implemented is shown. Figure 1 As shown, a host machine can run on computing environment 100, and one or more virtual machines can run on the host machine. For clarity, Figure 1 Only one virtual machine 101 is shown; however, it should be understood that multiple virtual machines can be configured in the computing environment 100. The computing environment 100 can be used to form a cloud computing platform powered by GPUs.

[0019] Application 102 can be developed based on various deep learning frameworks (e.g., TensorFlow) and can be used to call GPU resources at runtime. For example, when executed, application 102 can send a request to the underlying architecture to call GPU resources.

[0020] In some embodiments, computing environment 100 may be deployed based on the Unified Computing Device Architecture (CUDA) framework. However, it should be understood that the CUDA framework is provided as an example only, and the concepts of this disclosure can be applied to any other suitable framework currently known or developed in the future. The CUDA framework provides two programming interfaces: a runtime API and a driver API, where the runtime API is a high-level API built on top of the low-level API (driver API). The driver API provides control interfaces for CUDA contexts (similar to the concept of processes) and CUDA modules (similar to dynamic libraries). Because the runtime API is easier to use, upper-layer applications typically use the runtime API for programming. For example, application 102 may be a CUDA-based application and may be programmed using the runtime API.

[0021] like Figure 1 As shown, application 102 can call the runtime API during execution, and then call the driver API. Typically, the API defines a large number of library functions. When calling the API, application 102 usually specifies the name of the interface or library function to be called and its corresponding parameters. When calling the driver API, information such as the CUDA context can also be specified.

[0022] The pseudo-library 104 runs in user space of the virtual machine 101. Its library functions can have the same names as those in the real library and can be used to replace the real library files. In this way, the application 102 can be code programmed for a regular computing platform without needing to be adapted for environment 100. The pseudo-library 104 can intercept information or data related to requests from the application 102 and can package this information into a data packet in a specific format. This information may include the name of the called interface, parameters, and CUDA context, etc. The pseudo-library 104 passes this data to the front-end driver 106 via the I / O control interface. When the front-end driver 106 returns a value to the pseudo-library 104 via the I / O control interface, the pseudo-library 104 can obtain the return value of the CUDA interface.

[0023] By intercepting interface call requests in the user space of the virtual machine, developers can develop independently of hardware vendors, thereby significantly reducing development costs.

[0024] GPU sharing requires dynamic module loading and context switching, operations that can only be performed at the driver API layer. Therefore, in some embodiments, the pseudo-library 104 can intercept the driver API to achieve isolation of CUDA applications between virtual machines.

[0025] The front-end driver 106 runs in the kernel mode of the virtual machine and interacts with the pseudo-library 104 running in user mode via the I / O control interface. For example, the front-end driver 106 could be a Virtio driver. When the front-end driver 106 is invoked, it copies data from user mode to kernel mode, fills this data into a queue (e.g., a circular buffer), and sends the corresponding request to the back-end driver 108 on the host. After the back-end driver 108 processes the request, it fills the processing result into another queue (e.g., a circular buffer) and notifies the front-end driver 106 that processing is complete. Different virtual machines place their corresponding data in the same queue for processing by the back-end driver 108, thereby achieving GPU resource sharing.

[0026] The backend driver (or backend device emulator) 108 can share the virtual machine's memory through a queue or circular buffer mechanism, enabling zero-copy data communication between the frontend and backend, thereby improving performance. After receiving a request from the frontend driver 106, the backend driver 108 retrieves the corresponding data packet from the queue and parses it to obtain the name, parameters, CUDA context, etc., of the interface to be executed. Then, the backend driver 108 calls the actual interface to complete the processing via the physical GPU 110.

[0027] In some embodiments, the front-end driver 106 and the back-end driver 108 can be implemented using Virtio. Virtio is a semi-virtualization scheme in which a circular buffer is implemented using Vring to achieve zero-copy data.

[0028] Figure 2 A flowchart of a method 200 for sharing dedicated processing resources according to some embodiments of the present disclosure is shown. For example, the dedicated processing resources may be GPU resources. Method 200 may, for example, be at least partially... Figure 1 The method is executed in the computing environment 100. It should be understood that the method 200 can also be executed by any other suitable computing device or cloud server.

[0029] At step 202, in response to receiving a request to invoke an interface for a dedicated processing resource, information associated with the request is extracted in the user space of the virtual machine. For example, after receiving a call request from application 102, pseudo-library 104 can extract information associated with the request. In some embodiments, this information may include request parameters, the name of the interface, and the context of the request. For example, in a CUDA architecture, this information may include the name of the invoked interface, parameters, and the CUDA context.

[0030] In some embodiments, the invoked interface may be a driver API for dedicated processing resources. Compared to runtime APIs, driver APIs can extract richer information, such as context information.

[0031] At 204, in the kernel mode of the virtual machine, this information is provided to the host running the virtual machine. For example, the front-end driver 106 can provide this information to the back-end driver 108 on the host. In some embodiments, in the user mode of the virtual machine 101, the pseudo-library 104 can package this information into a data packet and forward the data packet to the host.

[0032] In some embodiments, this information can be populated in a queue, and a notification can be sent to the host so that the host can read the information from the queue after receiving the notification from the virtual machine. This queue can be a circular buffer, and in a Virtio-based architecture, it can be a Vring.

[0033] At 206, on the host, an interface for a dedicated processing resource (e.g., physical GPU 110) is invoked based on this information. In some embodiments, in response to receiving the execution result of the invocation of the interface from GPU 110, the host stores the execution result in another queue and sends a notification to virtual machine 101 to indicate that processing has been completed. In the kernel mode of virtual machine 101, the execution result is read from the queue and sent to application 102.

[0034] According to embodiments of this disclosure, multiple virtual machines can share access to the same dedicated processing resource device, such as the same physical GPU card, thereby improving the resource utilization of the cloud platform. Furthermore, by intercepting interface parameters and other information in the user space of the virtual machine, the virtual machine's context information can be better utilized, making the system more developer-friendly and easier to implement.

[0035] Figure 3 A schematic diagram of an apparatus 300 for sharing dedicated processing resources according to an embodiment of the present disclosure is shown. In some embodiments, the dedicated processing resource may be a graphics processing unit (GPU). The apparatus 300 may be composed of... Figure 1 The computing environment 100 shown is used to implement this.

[0036] like Figure 3 As shown, the apparatus 300 includes an extraction module 302 configured to extract information associated with a request in user space of a virtual machine in response to receiving a request for invoking an interface for a dedicated processing resource. In some embodiments, the information includes at least one of the following: parameters of the request, the name of the interface, and the context of the request.

[0037] In some embodiments, the interface includes a driver application programming interface for the dedicated processing resource.

[0038] The apparatus 300 includes a providing module 304 configured to provide the information to the host running the virtual machine in kernel mode of the virtual machine.

[0039] In some embodiments, the apparatus 300 may further include: a packaging module configured to package the information into a data packet in the user space of the virtual machine, and wherein the providing module 304 may include: a forwarding module configured to forward the data packet to the host.

[0040] In some embodiments, the providing module 304 may include: a filling module configured to fill the information into a queue; and a notification sending module configured to send a notification to the host, such that the host reads the information from the queue in response to receiving a notification from the virtual machine.

[0041] The device 300 includes a calling module 306 configured to, on the host, call the interface of the dedicated processing resource based on the information.

[0042] In some embodiments, the apparatus 300 further includes: a storage module configured on the host to, in response to receiving an execution result of calling the interface, store the execution result in a queue and send a notification to the virtual machine to indicate that processing has been completed; and a reading module configured to, in the kernel mode of the virtual machine, read the execution result from the queue.

[0043] Figure 4 A schematic block diagram of a device 400 that can be used to implement embodiments of the present disclosure is shown. Figure 1 The computing environment shown is 100 or as follows Figure 3 The device 300 shown can be implemented by the device 400. For example... Figure 4 As shown, device 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 402 or loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. CPU 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0044] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0045] The various processes and procedures described above, such as method 200, may be executed by processing unit 401. For example, in some embodiments, method 200 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by CPU 401, one or more steps of method 200 described above may be performed. Alternatively, in other embodiments, CPU 401 may be configured to execute method 200 by any other suitable means (e.g., by means of firmware).

[0046] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0047] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0048] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0049] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0050] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0051] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0052] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0053] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0054] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for sharing a dedicated processing resource, comprising: extracting, in response to receiving a request for invoking a driver application programming interface of a dedicated processing resource, information associated with the request through a pseudo library of a virtual machine in a user mode of the virtual machine, the information including at least one of parameters of the request, a name of the interface, and a context of the request; packing, in the user mode of the virtual machine, the information into a packet through the pseudo library; delivering, in the user mode of the virtual machine, the packet to a front-end driver of the virtual machine through the pseudo library; providing, in a kernel mode of the virtual machine, the packet to a host running the virtual machine through the front-end driver; and invoking, in the host, the driver application programming interface of the dedicated processing resource based on the packet. forwarding the packet to the host.

2. The method of claim 1, wherein providing the information to the host comprises: 3.The method of claim 1, wherein the providing the packet to the host comprises: stuffing the packet in a queue; and sending a notification to the host to cause the host to read the packet from the queue in response to receiving the notification from the virtual machine. 4.The method of claim 1, further comprising: storing, in the host, an execution result of invoking the interface in response to receiving the execution result, in a queue and sending a notification to the virtual machine to indicate that processing has been completed; and reading, in the kernel mode of the virtual machine, the execution result from the queue. 5.The method of claim 1, wherein the dedicated processing resource comprises a graphics processing unit (GPU). 6.An apparatus for sharing a dedicated processing resource, comprising: an extracting module configured to extract, in response to receiving a request for invoking a driver application programming interface of a dedicated processing resource, information associated with the request through a pseudo library of a virtual machine in a user mode of the virtual machine, the information including at least one of parameters of the request, a name of the interface, and a context of the request; a packing module configured to pack, in the user mode of the virtual machine, the information into a packet through the pseudo library; a delivering module configured to deliver, in the user mode of the virtual machine, the packet to a front-end driver of the virtual machine through the pseudo library; a providing module configured to provide, in a kernel mode of the virtual machine, the packet to a host running the virtual machine through the front-end driver; and an invoking module configured to invoke, in the host, the driver application programming interface of the dedicated processing resource based on the packet. 7.The apparatus of claim 6, wherein the providing module comprises: a forwarding module configured to forward the packet to the host. 8.The apparatus of claim 6, wherein the providing module comprises: a stuffing module configured to stuff the packet in a queue; and a notification sending module configured to send a notification to the host to cause the host to read the packet from the queue in response to receiving the notification from the virtual machine. ​ ​ ​ ​ ​ 9. The apparatus of claim 6, further comprising: a storage module configured to store, at the host, in response to receiving an execution result of a call to the interface, the execution result in a queue and send a notification to the virtual machine to indicate that processing has been completed; and a read module configured to read, at the kernel space of the virtual machine, the execution result from the queue.

10. The apparatus of claim 6, wherein the dedicated processing resource comprises a graphics processing unit (GPU).

11. An electronic device, the electronic device comprising: one or more processors; and memory storing one or more programs, when executed by the one or more processors, cause the electronic device to implement the method of any of claims 1-5.

12. A computer-readable storage medium having stored thereon a computer program, the program, when executed by a processor, implementing the method of any of claims 1-5.

13. A computer program product comprising a computer program, the computer program, when executed by a processor, implementing the method of any of claims 1-5. ​ ​

Images