CUDA (Compute Unified Device Architecture) resource acquisition method and device, equipment, medium and product

Abstract

The invention discloses a CUDA (Compute Unified Device Architecture) resource acquisition method, device and equipment, a medium and a product. The method comprises: receiving a resource request sent by an intermediate device, the resource request being a resource request sent by a client to middleware, and the resource request carrying a target server address; intercepting a resource request through the vCUDA interface, and sending the resource request to a VCUDA library; analyzing the resource request through a VCUDA library to obtain a resource request type; determining a target CUDA according to the resource request type and the resource request; the resource request is sent to the target CUDA to obtain CUDA resources, and by means of the technical scheme, the problems that virtualization is carried out on a GPU drive, virtualization CUDA support is provided in a software full virtualization mode, consequently, processing complexity and time cost are increased, expenditure is large, and the scheduling capacity is usually limited by virtualization software can be solved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a CUDA resource acquisition method, device, equipment, medium and product. Background Art

[0002] The training of deep learning models requires strong computing power support. Computing power is the ability to process data and has become the key to the development of deep learning models. Traditional computing power faces challenges. CPUs cannot meet the needs of big data parallel scenarios. GPUs (Graphics Processing Units) have gradually become the main supporter of computing power due to their powerful parallel processing capabilities.

[0003] CUDA (Compute Unified Device Architecture) is a general-purpose parallel computing architecture that enables GPUs to solve complex computing problems.

[0004] For GPUs with powerful computing power, if some small AI inference tasks monopolize the GPU resources, it will cause resource waste. This requires dividing the GPU into virtual slices, and then multiple tasks share the same GPU resources to fully utilize the GPU computing resources.

[0005] The above method is used to implement GPU resource scheduling, which is to perform virtualization on top of the GPU driver and provide virtualized CUDA support through software full virtualization. This additional software layer needs to handle the interaction between the virtual machine and the physical hardware, which increases the complexity and time cost of the processing. The overhead is large, and the scheduling capability is usually limited by the virtualization software, such as fixed vGPU configuration, fixed resource allocation ratio, etc. Summary of the invention

[0006] Embodiments of the present invention provide a CUDA resource acquisition method, apparatus, device, medium and product to solve at least one of the above problems.

[0007] According to one aspect of the present invention, a CUDA resource acquisition method is provided, which is executed by a target server, wherein the target server includes: a local server or a remote server, and the CUDA resource acquisition method includes:

[0008] Receiving a resource request sent by an intermediate device, wherein the resource request is a resource request sent by a client to a middleware, and the resource request carries a target server address;

[0009] Intercept resource requests through the vCUDA interface and send the resource requests to the VCUDA library;

[0010] Parsing the resource request through the VCUDA library to obtain the resource request type;

[0011] Determine the target CUDA based on the resource request type and resource request;

[0012] Send resource requests to the target CUDA to obtain CUDA resources.

[0013] According to another aspect of the present invention, a CUDA resource acquisition method is provided, which is executed by an intermediate device, and the CUDA resource acquisition method includes:

[0014] Receive a resource request sent by a client, wherein the resource request carries a target server address;

[0015] The resource request is sent to the target server so that the target server intercepts the resource request through the vCUDA interface, and the resource request is sent to the VCUDA library. The resource request is parsed by the VCUDA library to obtain the resource request type, and the target CUDA is determined according to the resource request type and the resource request, and the resource request is sent to the target CUDA to obtain CUDA resources.

[0016] According to another aspect of the present invention, a CUDA resource request device is provided, which is configured in a target server, wherein the target server includes: a local server or a remote server, and the CUDA resource request device includes:

[0017] A first resource request receiving module, configured to receive a resource request sent by an intermediate device, wherein the resource request is a resource request sent by a client to a middleware, and the resource request carries a target server address;

[0018] A first resource request sending module, used for intercepting resource requests through a vCUDA interface and sending the resource requests to a VCUDA library;

[0019] A resource request parsing module, used to parse the resource request through the VCUDA library to obtain the resource request type;

[0020] A target CUDA determination module, used for determining a target CUDA according to a resource request type and a resource request;

[0021] The CUDA resource acquisition module is used to send a resource request to the target CUDA to obtain CUDA resources.

[0022] According to another aspect of the present invention, a CUDA resource request device is provided, which is configured in an intermediate device, and the CUDA resource request device includes:

[0023] A second resource request receiving module, configured to receive a resource request sent by a client, wherein the resource request carries a target server address;

[0024] The second resource request sending module is used to send the resource request to the target server, so that the target server intercepts the resource request through the vCUDA interface, sends the resource request to the VCUDA library, parses the resource request through the VCUDA library, obtains the resource request type, determines the target CUDA according to the resource request type and the resource request, and sends the resource request to the target CUDA to obtain CUDA resources.

[0025] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0026] at least one processor; and

[0027] a memory communicatively connected to the at least one processor; wherein,

[0028] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the CUDA resource acquisition method described in any embodiment of the present invention.

[0029] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the CUDA resource acquisition method described in any embodiment of the present invention when executed.

[0030] According to another aspect of the present invention, a computer program product is provided. When the computer program is executed by a processor, the computer program implements the CUDA resource acquisition method as described in any one of the embodiments of the present invention.

[0031] The embodiment of the present invention sends a resource request to an intermediate device through a client, and the intermediate device sends the resource request to the target server according to the address of the target server carried in the resource request, and the target server receives the resource request; intercepts the resource request through the vCUDA interface, and sends the resource request to the VCUDA library; parses the resource request through the VCUDA library to obtain the resource request type; determines the target CUDA according to the resource request type and the resource request; sends the resource request to the target CUDA to obtain CUDA resources, which solves the problem that virtualization is performed on the GPU driver, and virtualized CUDA support is provided through software full virtualization, which increases the complexity and time cost of processing, has a large overhead, and the scheduling capability is usually limited by the virtualization software, and improves the scheduling and allocation capabilities of CUDA resources.

[0032] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 is a flow chart of a CUDA resource acquisition method in an embodiment of the present invention;

[0035] Figure 2 is a comparison diagram of virtualization in an embodiment of the present invention and virtualization in the prior art;

[0036] Figure 3 It is a schematic diagram of a client making a resource request when the target server is a local server in an embodiment of the present invention;

[0037] Figure 4 It is a schematic diagram of resource request when the target server is a local server and the target CUDA fails or is fully loaded in an embodiment of the present invention;

[0038] Figure 5 It is a schematic diagram of a resource request process when the target server is a remote server in an embodiment of the present invention;

[0039] Figure 6 It is a schematic diagram of a resource request process when the target server is a remote server in an embodiment of the present invention and a CUDA failure or full load condition is detected.

[0040] Figure 7 is a flowchart of another CUDA resource acquisition method in an embodiment of the present invention;

[0041] Figure 8 is a structural schematic diagram of a CUDA resource acquisition device in an embodiment of the present invention;

[0042] Fig. 9 is a structural schematic diagram of another CUDA resource acquisition device in an embodiment of the present invention;

[0043] Fig.10 It is a structural schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0047] Embodiment 1

[0048] Figure 1 A flowchart of a CUDA resource acquisition method provided in an embodiment of the present invention is provided. This embodiment is applicable to the case of CUDA resource acquisition. The method can be executed by a CUDA resource acquisition device in an embodiment of the present invention. The device can be implemented in software and / or hardware. Figure 1 As shown, the method specifically comprises the following steps:

[0049] S110: Receive a resource request sent by an intermediate device.

[0050] It should be noted that the CUDA resource acquisition method provided in an embodiment of the present invention is applied to a CUDA resource acquisition system, and the CUDA resource acquisition system includes: at least one client, an intermediate device, at least one local server, and at least one remote server. The client is connected to the intermediate device, and the intermediate device is respectively connected to the client, the local server, and the remote server. The client user sends a resource request, and the intermediate device is used to receive the resource request sent by the client. If the target server address carried by the resource request is the local server address, the intermediate device sends the resource request to the local server. If the target server address carried by the resource request is the remote server address, the intermediate device sends the resource request to the remote server. The local server includes: a vCUDA interface, a VCUDA library, a CUDA forwarder, CUDA, and a GPU. The remote server includes: a vCUDA interface, a VCUDA library, a CUDA forwarder, CUDA, and a GPU.

[0051] The resource request is a resource request sent by the client to the middleware, and the resource request carries the target server address. The resource request is a request for requesting CUDA resources.

[0052] It should be noted that if the target server address carried in the resource request is a local server address, the target server is the local server; if the target server address carried in the resource request is a remote server address, the target server is the remote server.

[0053] Specifically, the client sends a resource request to the intermediate device. After receiving the resource request sent by the client, the intermediate device determines whether to send the resource request to the local server or the remote server according to the target server address carried in the resource request. If the target server address is the local server address, the resource request is sent to the local server. If the target server address is the remote server address, the resource request is sent to the remote server.

[0054] S120, intercepting the resource request through the vCUDA interface, and sending the resource request to the VCUDA library.

[0055] It should be noted that the embodiment of the present invention performs virtualization on the CUDA library driver API layer by abstracting the CUDA library, and supports multi-user dynamic concurrency by dynamically creating virtual handles. The CUDA library is a set of function definitions whose prototypes are described in the header file, and all calculations implemented by the CUDA library use GPU acceleration.

[0056] There may be multiple VCUDA libraries at the target server, and the resource request may be sent to any VCUDA library, or may be sent to corresponding VCUDA libraries in sequence according to the VCUDA library sequence. This is not limited in the embodiment of the present invention.

[0057] Specifically, if the target server is a local server, the resource request is intercepted through the vCUDA interface of the local server, and the resource request is sent to the VCUDA library of the local server. If the target server is a remote server, the resource request is intercepted through the vCUDA interface of the remote server, and the resource request is sent to the VCUDA library of the remote server.

[0058] S130, parsing the resource request through the VCUDA library to obtain a resource request type.

[0059] Among them, the resource request type includes: a first type and a second type. If the attribute information carried by the resource request only includes: a resource request amount, then the resource request type is determined to be the first type. If the attribute information carried by the resource request includes: a resource request amount and a GPU identifier, then the resource request type is determined to be the second type.

[0060] Optionally, the resource request is parsed through a VCUDA library to obtain a resource request type, including:

[0061] Parsing the resource request through the VCUDA library to obtain attribute information carried in the resource request;

[0062] If the attribute information includes: the resource request amount, determining the resource request type as the first type;

[0063] If the attribute information includes: the resource request amount and the GPU identifier, the resource request type is determined to be the second type.

[0064] The resource request amount may be a CUDA resource request amount.

[0065] Specifically, if the attribute information only includes the resource request amount, the resource request type is determined to be the first type; if the attribute information includes: the resource request amount and the GPU identifier, the resource request type is determined to be the second type.

[0066] S140, determining a target CUDA according to the resource request type and the resource request.

[0067] The target CUDA may be the CUDA of a local server or the CUDA of a remote server. If the target server is a local server, the target CUDA is the CUDA of the local server. If the target server is a remote server, the target CUDA is the CUDA of the remote server.

[0068] Specifically, the method of determining the target CUDA according to the resource request type and the resource request can be: if the resource request type is the first type, the CUDAs in multiple servers are screened according to the resource occupancy carried by the resource request and the resource occupancy of multiple servers to obtain the target CUDA; if the resource request type is the second type, the CUDA corresponding to the GPU identifier carried by the resource request is determined as the target CUDA.

[0069] Optionally, the target CUDA is determined based on the resource request type and the resource request, including:

[0070] If the resource request type is the first type, CUDAs in the multiple servers are screened according to the resource occupancy carried in the resource request and the resource occupancy of the multiple servers to obtain a target CUDA;

[0071] If the resource request type is the second type, the CUDA corresponding to the GPU identifier carried in the resource request is determined as the target CUDA.

[0072] It should be noted that when the target server is a local server, if the resource request type is the first type, the CUDAs in multiple local servers are screened according to the resource occupancy carried by the resource request and the resource occupancy of multiple local servers to obtain the target CUDA; when the target server is a remote server, if the resource request type is the first type, the CUDAs in multiple remote servers are screened according to the resource occupancy carried by the resource request and the resource occupancy of multiple remote servers to obtain the target CUDA.

[0073] In a specific example, if the resource request type is the second type, the resource request is sent through the vCUDA forwarder to the CUDA library of the same server as the GPU corresponding to the GPU identifier to obtain the CUDA resource.

[0074] In another specific example, when the target server is a local server, if the resource request type is the first type, the resource request can be sent to any local server to access the CUDA library to obtain CUDA resources, and then horizontal expansion can be achieved through load balancing and other means.

[0075] In another specific example, when the target server is a remote server, if the resource request type is the first type, the resource request can be sent to any remote server to access the CUDA library to obtain CUDA resources, and then horizontal expansion can be achieved through load balancing and other means.

[0076] S150, sending a resource request to the target CUDA to obtain CUDA resources.

[0077] Specifically, the method of sending a resource request to the target CUDA to obtain CUDA resources can be: if the resource request is of the first type, the resource request is sent to the target CUDA through the VCUDA library to obtain CUDA resources; if the resource request is of the second type, the resource request is sent to the target CUDA through the vCUDA forwarder to obtain CUDA resources. The method of sending a resource request to the target CUDA to obtain CUDA resources can also be: if the target CUDA is operating normally, the resource request is sent to the target CUDA to obtain CUDA resources; if the target CUDA fails or is fully loaded, and the resource request type is of the first type, the resource request is sent to the CUDA corresponding to another GPU of the current server, or the CUDA corresponding to the GPU of another server, through the vCUDA forwarder to obtain CUDA resources; if the target CUDA fails or is fully loaded, and the resource request type is of the second type, the resource request is sent to the CUDA corresponding to another GPU of the current server through the vCUDA forwarder to obtain CUDA resources.

[0078] In a specific example, after receiving a resource request sent by an intermediate device, the local server is used to intercept the resource request through the vCUDA interface in the local server, send the resource request to the VCUDA library in the local server, parse the resource request through the VCUDA library in the local server, obtain the resource request type, determine the target CUDA according to the resource request type and the resource request, and the local server sends the resource request to the target CUDA to obtain CUDA resources.

[0079] In another specific example, after receiving the resource request sent by the intermediate device, the remote server is used to intercept the resource request through the vCUDA interface in the remote server, send the resource request to the VCUDA library in the remote server, parse the resource request through the VCUDA library in the remote server to obtain the resource request type, determine the target CUDA according to the resource request type and the resource request, and the remote server sends the resource request to the target CUDA to obtain CUDA resources.

[0080] In another specific example, Figure 2 As shown, in the prior art, GPU is virtualized to fully utilize GPU resources. Virtualizing GPU increases the complexity and time cost of processing, has high overhead, and the scheduling capability is usually limited by the virtualization software. The embodiment of the present invention virtualizes CUDA, which can improve system efficiency while solving the above problems.

[0081] Optionally, a resource request is sent to the target CUDA to obtain CUDA resources, including:

[0082] If the resource request is of the first type, the resource request is sent to the target CUDA through the VCUDA library to obtain CUDA resources;

[0083] If the resource request is of the second type, the resource request is sent to the target CUDA through the vCUDA forwarder to obtain CUDA resources.

[0084] Specifically, if the resource request is of the first type and the target server is a local server, the resource request is sent through the VCUDA library to any local server to access the CUDA library to obtain CUDA resources. If the resource request is of the second type and the target server is a local server, the resource request is sent through the vCUDA forwarder to the CUDA library of the same server to obtain CUDA resources. If the resource request is of the first type and the target server is a remote server, the resource request is sent through the VCUDA library to any remote server to access the CUDA library to obtain CUDA resources. If the resource request is of the second type and the target server is a remote server, the resource request is sent through the vCUDA forwarder to the CUDA library of the same server to obtain CUDA resources.

[0085] In another specific example, Figure 3 A schematic diagram of a client making a resource request, such as Figure 3 As shown, the client sends a resource request to the intermediate device through a container or a virtual machine. The intermediate device sends the resource request to the local server (not shown in the figure) according to the target server address carried in the resource request, which is the local server. After receiving the resource request, the local server intercepts the resource request through the vCUDA interface and sends it to the vCUDA library; the vCUDA library parses the resource request. If the intercepted resource request is a stateful request (the second type), the vCUDA forwarder needs to send the request to the CUDA library of the same server to obtain CUDA resources; if the intercepted resource request is stateless (the first type), for the vCUDA library, the request can be sent to any local server to access the CUDA library to obtain CUDA resources. Horizontal expansion can be achieved through load balancing and other means.

[0086] Optionally, a resource request is sent to the target CUDA to obtain CUDA resources, including:

[0087] If the target CUDA runs normally, a resource request is sent to the target CUDA to obtain CUDA resources;

[0088] If the target CUDA fails or is fully loaded, and the resource request type is the first type, the resource request is sent to the CUDA corresponding to another GPU of the current server, or the CUDA corresponding to the GPU of another server, through the vCUDA forwarder, to obtain CUDA resources;

[0089] If the target CUDA fails or is fully loaded, and the resource request type is the second type, the resource request is sent to the CUDA corresponding to another GPU of the current server through the vCUDA forwarder to obtain CUDA resources.

[0090] It should be noted that if the target CUDA does not fail and the target CUDA is not fully loaded, it is determined that the target CUDA is operating normally.

[0091] The other GPU of the current server may be a CUDA corresponding to another GPU in the current server that is operating normally except the GPU corresponding to the target CUDA. The CUDA corresponding to the other GPU of the current server has not failed and is not fully loaded.

[0092] Specifically, if the target CUDA fails or is fully loaded, and the resource request type is the first type, the resource request is sent to the CUDA corresponding to another GPU of the current server, or the CUDA corresponding to the GPU of another server, and the method for obtaining CUDA resources can be: when the target server is a local server, if the target CUDA fails or is fully loaded, and the resource request type is the first type, the resource request is sent to the CUDA corresponding to another GPU of the current server, or the CUDA corresponding to the GPU of another local server, through the vCUDA forwarder, to obtain CUDA resources. When the target server is a remote server, if the target CUDA fails or is fully loaded, and the resource request type is the first type, the resource request is sent to the CUDA corresponding to another GPU of the current server, or the CUDA corresponding to the GPU of another remote server, through the vCUDA forwarder, to obtain CUDA resources.

[0093] In another specific example, Figure 4 This is a schematic diagram of resource requests when the target CUDA fails or is fully loaded, such as Figure 4As shown, the client sends a resource request to the intermediate device through a container / virtual machine. The intermediate device sends the resource request to the local server according to the target server address carried in the resource request as the local server (not shown in the figure). After the local server receives the resource request, it intercepts the resource request through the vCUDA interface and sends it to the vCUDA library; the vCUDA library parses the resource request. If the intercepted resource request is a stateful request (the second type), the vCUDA forwarder needs to send the request to the CUDA library of the same server to obtain CUDA resources. However, if the CUDA of the local GPU card accessed at this time is faulty or fully loaded, the CUDA library of another local GPU card is dynamically dispatched to obtain CUDA resources. If the intercepted resource request is stateless (the first type), for the vCUDA library, the request can be sent to any local server to access the CUDA library to obtain CUDA resources, and horizontal expansion can be achieved through load balancing and other means; similarly, if the CUDA of the local GPU card accessed is faulty or fully loaded, the CUDA library of another local GPU card is dynamically dispatched to obtain CUDA resources.

[0094] Optionally, the target server is a remote server;

[0095] Intercept resource requests through the vCUDA interface and send resource requests to the VCUDA library, including:

[0096] The resource requests of multiple clients sent by the middleware are intercepted through the vCUDA interface of the remote server, and the resource requests sent by the multiple clients are sent to the VCUDA library in sequence according to the priorities of the resource requests of the multiple clients.

[0097] Specifically, the priority of the client's resource request can be determined by: presetting the priorities of multiple clients, and determining the priority of the client as the priority of the resource request sent by the client; or by presetting the priorities corresponding to multiple resource request types, obtaining the resource request types corresponding to the resource requests sent by multiple clients, and then determining the priority of the client's resource request according to the priority of the resource request type corresponding to the resource request sent by the client. It can also be: determining the priority of the client's resource request according to the amount of resource requests carried by the resource requests of multiple clients. For example, the larger the amount of resource requests, the higher the priority of the resource request.

[0098] It should be noted that resource requests with high priority are pre-processed. For example, if the priorities of resource requests from multiple clients are determined based on the resource request amounts carried by the resource requests from multiple clients, resource requests with large resource request amounts are processed first.

[0099] In a specific example, Figure 5 When the target server is a remote server, the resource request process diagram is as follows: Figure 5 As shown in the figure, the multi-node of the distributed system includes multi-node request and multi-node forwarding processing. Multiple clients can send requests to the remote server. When the amount of resource requests increases, the multi-resource requests can be distributed to different nodes for forwarding processing. The resource request processing process in the multi-node of the distributed system is as follows: the local server can use the CUDA library of its GPU card in combination with the remote server through RDMA high performance. In the distributed system, the client sends a resource request to the intermediate device through a container / virtual machine. The intermediate device sends the resource request to the local server through RDMA according to the target server address carried by the resource request as the remote server (not shown in the figure). The resource request is intercepted by the vCUDA interface of the remote server and sent to the vCUDA library; the vCUDA library parses the user request. If the intercepted resource request is a stateful request, the vCUDA forwarder sends the resource request to the CUDA library of the same server to obtain CUDA resources; if the intercepted resource request is stateless, then for the vCUDA library, the request can be sent to any remote server to access the CUDA library to obtain CUDA resources, and horizontal expansion can be achieved through load balancing and other means.

[0100] The embodiments of the present invention combine high-performance RDMA to perform remote communication in a distributed environment, thereby improving the global scheduling and allocation capabilities of CUDA resources in the distributed environment and improving system efficiency.

[0101] In a specific example, Figure 6 When the target server is a remote server, the resource request process diagram is marked with CUDA failure or full load, such as Figure 6As shown, the client sends a resource request to the intermediate device through a container / virtual machine. The intermediate device sends the resource request to the local server through RDMA based on the target server address carried in the resource request as the remote server (not shown in the figure). The resource request is intercepted by the vCUDA interface of the remote server and sent to the vCUDA library; the vCUDA library parses the user request. If the intercepted resource request is a stateful request, the vCUDA forwarder sends the resource request to the CUDA library of the same server to obtain CUDA resources; if there is a fault or full load when accessing the CUDA of this remote GPU card, the CUDA library of another GPU card of this server node is dynamically dispatched to obtain CUDA resources; if the intercepted resource request is stateless, then for the vCUDA library, the request can be sent to any remote server to access the CUDA library to obtain CUDA resources; similarly, if there is a fault or full load when accessing the CUDA of this GPU card, the CUDA library of the GPU card of another remote server node of this server or this distributed system can be dynamically dispatched to obtain CUDA resources.

[0102] The technical solution of this embodiment is that the client sends a resource request to the intermediate device, and the intermediate device sends the resource request to the target server according to the address of the target server carried in the resource request, and the target server receives the resource request; the resource request is intercepted by the vCUDA interface, and the resource request is sent to the VCUDA library; the resource request is parsed by the VCUDA library to obtain the resource request type; the target CUDA is determined according to the resource request type and the resource request; the resource request is sent to the target CUDA to obtain CUDA resources, which solves the problem of virtualization on the GPU driver, providing virtualized CUDA support through software full virtualization, which increases the complexity and time cost of processing, has a large overhead, and the scheduling capability is usually limited by the virtualization software, and improves the scheduling and allocation capabilities of CUDA resources.

[0103] Embodiment 2

[0104] Figure 7 A flowchart of a CUDA resource acquisition method provided in an embodiment of the present invention is provided. This embodiment is applicable to the case of CUDA resource acquisition. The method can be executed by a CUDA resource acquisition device in an embodiment of the present invention. The device can be implemented in software and / or hardware. Figure 7 As shown, the method specifically comprises the following steps:

[0105] S710: Receive a resource request sent by a client.

[0106] Wherein, the resource request carries the target server address.

[0107] S720, sending the resource request to the target server so that the target server intercepts the resource request through the vCUDA interface, sending the resource request to the VCUDA library, parsing the resource request through the VCUDA library to obtain the resource request type, determining the target CUDA according to the resource request type and the resource request, and sending the resource request to the target CUDA to obtain CUDA resources.

[0108] It should be noted that if the target server address carried in the resource request is a local server address, the resource request is sent to the local server; if the target server address carried in the resource request is a remote server address, the resource request is sent to the remote server.

[0109] The processing process of the local server after receiving the resource request is the same as that in the above embodiment, which will not be repeated here. The processing process of the remote server after receiving the resource request is the same as that in the above embodiment, which will not be repeated here.

[0110] Optionally, receive a resource request sent by a client, including:

[0111] Receive resource requests sent by clients through containers or virtual machines.

[0112] The embodiments of the present invention natively provide CUDA virtualization support for a mixed environment of virtual machines and containers in virtualization technology.

[0113] It should be noted that the embodiment of the present invention performs virtualization on the CUDA library interface layer, and proposes to build a unified virtual layer management method to support the unified use and management of a mixed environment of virtual machines and containers; and combines high-performance RDMA networks to efficiently support in a distributed environment, and utilizes a combination of multi-node requests and multi-node processing to improve the global scheduling and allocation capabilities of CUDA resources in a distributed environment, thereby improving system efficiency.

[0114] The embodiment of the present invention adopts a distributed cluster as the overall implementation, deploys virtual machines and docker containers as CUDA virtualization experimental carriers. In specific applications, dynamic concurrency of multiple users is supported by dynamically creating virtual handles. The embodiment of the present invention can meet the simultaneous management of virtual machines and container virtualization, natively provide virtualization support for virtual machines and container technologies in virtualization technology, and use RDMA to perform efficient remote communication.

[0115] The technical solution of this embodiment is that the client sends a resource request to the intermediate device, and the intermediate device sends the resource request to the target server according to the address of the target server carried in the resource request, and the target server receives the resource request; the resource request is intercepted by the vCUDA interface, and the resource request is sent to the VCUDA library; the resource request is parsed by the VCUDA library to obtain the resource request type; the target CUDA is determined according to the resource request type and the resource request; the resource request is sent to the target CUDA to obtain CUDA resources, which solves the problem of virtualization on the GPU driver, providing virtualized CUDA support through software full virtualization, which increases the complexity and time cost of processing, has a large overhead, and the scheduling capability is usually limited by the virtualization software, and improves the scheduling and allocation capabilities of CUDA resources.

[0116] Embodiment 3

[0117] Figure 8 A schematic diagram of the structure of a CUDA resource request device provided in an embodiment of the present invention. This embodiment is applicable to the case of CUDA resource request. The device can be implemented in software and / or hardware. The device can be integrated in any device that provides CUDA resource request function, such as a local server or a remote server. Figure 8 As shown, the CUDA resource request device specifically includes: a first resource request receiving module 810 , a first resource request sending module 820 , a resource request parsing module 830 , a target CUDA determining module 840 and a CUDA resource acquiring module 850 .

[0118] The first resource request receiving module is used to receive a resource request sent by an intermediate device, wherein the resource request is a resource request sent by a client to a middleware, and the resource request carries a target server address;

[0119] A first resource request sending module, used for intercepting resource requests through a vCUDA interface and sending the resource requests to a VCUDA library;

[0120] A resource request parsing module, used to parse the resource request through the VCUDA library to obtain the resource request type;

[0121] A target CUDA determination module, used for determining a target CUDA according to a resource request type and a resource request;

[0122] The CUDA resource acquisition module is used to send a resource request to the target CUDA to obtain CUDA resources.

[0123] The above-mentioned product can execute the method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0124] Embodiment 4

[0125] Fig. 9 The present invention provides a schematic diagram of the structure of a CUDA resource request device. This embodiment is applicable to the case of CUDA resource request. The device can be implemented in software and / or hardware. The device can be integrated in any device that provides CUDA resource request function, such as an intermediate device, such as Fig. 9 As shown, the CUDA resource request device specifically includes: a second resource request receiving module 910 and a second resource request sending module 920.

[0126] The second resource request receiving module is used to receive a resource request sent by a client, wherein the resource request carries a target server address;

[0127] The second resource request sending module is used to send the resource request to the target server, so that the target server intercepts the resource request through the vCUDA interface, sends the resource request to the VCUDA library, parses the resource request through the VCUDA library, obtains the resource request type, determines the target CUDA according to the resource request type and the resource request, and sends the resource request to the target CUDA to obtain CUDA resources.

[0128] The above-mentioned product can execute the method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0129] Embodiment 5

[0130] Fig.10 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0131] like Fig.10As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0132] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0133] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the CUDA resource acquisition method.

[0134] In some embodiments, the CUDA resource acquisition method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the CUDA resource acquisition method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the CUDA resource acquisition method in any other appropriate manner (e.g., by means of firmware).

[0135] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0136] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0137] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0138] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0139] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0140] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0141] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0142] An embodiment of the present invention further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the CUDA resource acquisition method according to any embodiment of the present invention.

[0143] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0144] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A CUDA resource acquisition method, characterized in that: The target server is executed by a target server, wherein the target server includes a local server or a remote server, and the CUDA resource acquisition method includes: Receiving a resource request sent by an intermediate device, wherein the resource request is a resource request sent by a client to a middleware, and the resource request carries a target server address; Intercept resource requests through the vCUDA interface and send the resource requests to the VCUDA library; Parsing the resource request through the VCUDA library to obtain the resource request type; Determine the target CUDA based on the resource request type and resource request; Send resource requests to the target CUDA to obtain CUDA resources.

2. The method according to claim 1, characterized in that Determine the target CUDA based on the resource request type and resource request, including: If the resource request type is the first type, CUDAs in the multiple servers are screened according to the resource occupancy carried in the resource request and the resource occupancy of the multiple servers to obtain a target CUDA; If the resource request type is the second type, the CUDA corresponding to the GPU identifier carried in the resource request is determined as the target CUDA.

3. The method according to claim 1, characterized in that Send resource requests to the target CUDA to obtain CUDA resources, including: If the target CUDA runs normally, a resource request is sent to the target CUDA to obtain CUDA resources; If the target CUDA fails or is fully loaded, and the resource request type is the first type, the resource request is sent to the CUDA corresponding to another GPU of the current server, or the CUDA corresponding to the GPU of another server, through the vCUDA forwarder, to obtain CUDA resources; If the target CUDA fails or is fully loaded, and the resource request type is the second type, the resource request is sent to the CUDA corresponding to another GPU of the current server through the vCUDA forwarder to obtain CUDA resources.

4. The method according to claim 1, characterized in that: The target server is a remote server; Intercept resource requests through the vCUDA interface and send resource requests to the VCUDA library, including: The resource requests of multiple clients sent by the middleware are intercepted through the vCUDA interface of the remote server, and the resource requests sent by the multiple clients are sent to the VCUDA library in sequence according to the priorities of the resource requests of the multiple clients.

5. The method according to claim 1, characterized in that The resource request is parsed through the VCUDA library to obtain the resource request type, including: Parsing the resource request through the VCUDA library to obtain attribute information carried in the resource request; If the attribute information includes: the resource request amount, determining the resource request type as the first type; If the attribute information includes: the resource request amount and the GPU identifier, the resource request type is determined to be the second type.

6. The method according to claim 1, characterized in that Send resource requests to the target CUDA to obtain CUDA resources, including: If the resource request is of the first type, the resource request is sent to the target CUDA through the VCUDA library to obtain CUDA resources; If the resource request is of the second type, the resource request is sent to the target CUDA through the vCUDA forwarder to obtain CUDA resources.

7. A CUDA resource acquisition method, characterized in that: Executed by an intermediate device, the CUDA resource acquisition method includes: Receive a resource request sent by a client, wherein the resource request carries a target server address; The resource request is sent to the target server so that the target server intercepts the resource request through the vCUDA interface, and the resource request is sent to the VCUDA library. The resource request is parsed by the VCUDA library to obtain the resource request type, and the target CUDA is determined according to the resource request type and the resource request, and the resource request is sent to the target CUDA to obtain CUDA resources.

8. The method according to claim 7, characterized in that Receive resource requests sent by clients, including: Receive resource requests sent by clients through containers or virtual machines.

9. A CUDA resource request device, characterized in that: Configured in a target server, the target server includes: a local server or a remote server, and the CUDA resource request device includes: A first resource request receiving module, configured to receive a resource request sent by an intermediate device, wherein the resource request is a resource request sent by a client to a middleware, and the resource request carries a target server address; A first resource request sending module, used for intercepting resource requests through a vCUDA interface and sending the resource requests to a VCUDA library; A resource request parsing module, used to parse the resource request through the VCUDA library to obtain the resource request type; A target CUDA determination module, used for determining a target CUDA according to a resource request type and a resource request; The CUDA resource acquisition module is used to send a resource request to the target CUDA to obtain CUDA resources.

10. A CUDA resource acquisition device, characterized in that: Configured in the intermediate device, the CUDA resource acquisition device includes: A second resource request receiving module, configured to receive a resource request sent by a client, wherein the resource request carries a target server address; The second resource request sending module is used to send the resource request to the target server, so that the target server intercepts the resource request through the vCUDA interface, sends the resource request to the VCUDA library, parses the resource request through the VCUDA library, obtains the resource request type, determines the target CUDA according to the resource request type and the resource request, and sends the resource request to the target CUDA to obtain CUDA resources.

11. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the CUDA resource acquisition method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the CUDA resource acquisition method according to any one of claims 1 to 8 when executed.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the CUDA resource acquisition method according to any one of claims 1 to 8.

Images