GPU resource scheduling method, device, equipment and storage medium

Abstract

The present application provides a GPU resource scheduling method, apparatus, device, and storage medium, which relate to the field of computer technology and can fully utilize GPU resources and avoid GPU resource conflicts. The method includes: obtaining an application resource description of an application, determining a target GPU that meets the application resource description; the application resource description is used to indicate the GPU capability requirements of the application; obtaining a start application request for starting the application, and starting the application; when the resource request of the application is obtained, if it is determined that the current available resources of the target GPU meet the resources indicated by the application resource description, allocating the resources indicated by the application resource description in the target GPU to the application; or, after obtaining the resource request of the application, in the process of the application using the target GPU resources, controlling the application to use target GPU resources that are less than or equal to the resources indicated by the application resource description. The present application can be used in the process of GPU resource scheduling.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a GPU resource scheduling method, apparatus, device, and storage medium. Background Art

[0002] With the rapid development of computer technology, graphics processing units (GPUs) have evolved into highly parallel, multi-threaded, and multi-core processors with powerful computing capabilities. With their powerful capabilities, GPUs have been widely used in popular research fields such as video transcoding, physics simulation, image processing, and network security.

[0003] How to fully utilize the computing power of GPU to solve complex computing problems and ensure that GPU resources do not conflict, that is, the research on GPU resource scheduling solutions, is one of the research hotspots in the field of computer technology today. Summary of the Invention

[0004] The present application provides a GPU resource scheduling method, apparatus, device, and storage medium, which can fully utilize GPU resources and avoid GPU resource conflicts.

[0005] In a first aspect, the present application provides a resource scheduling method based on a virtual graphics processor (GPU), the method comprising: obtaining an application resource description of an application, and determining a target GPU that meets the application resource description; the application resource description is used to indicate the GPU capability requirements of the application; obtaining a start application request for starting the application, and starting the application; when the resource request of the application is obtained, if it is determined that the current available resources of the target GPU meet the resources indicated by the application resource description, allocating the resources indicated by the application resource description in the target GPU to the application; or, after obtaining the resource request of the application, during the process of the application using the target GPU resources, controlling the application to use the target GPU resources so that they are less than or equal to the resources indicated by the application resource description.

[0006] In a possible implementation, there may be one or more target GPUs that meet the application resource description.

[0007] In one possible implementation, before starting the application, the method further includes: creating a virtual GPU corresponding to the application; upon obtaining the application's resource request, if it is determined that the currently available resources of the target GPU meet the resources indicated by the application resource description, allocating the resources indicated by the application resource description in the target GPU to the application, including: upon obtaining the application's resource request, if the virtual GPU determines that the currently available resources of the target GPU meet the resources indicated by the application resource description, allocating the resources indicated by the application resource description in the target GPU to the application; after obtaining the application's resource request, during the process of the application using the target GPU resources, controlling the application to use the target GPU resources to be less than or equal to the resources indicated by the application resource description, including: after obtaining the application's resource request, during the process of the application using the target GPU resources, controlling the application to use the target GPU resources to be less than or equal to the resources indicated by the application resource description.

[0008] In one possible implementation, the number of virtual GPUs corresponding to the created application is the same as the number of target GPUs that meet the application resource description of the application. A virtual GPU is used to control access to its corresponding target GPU.

[0009] In another possible implementation, the application resource description includes the rated video memory of the application; the resource request is used to allocate video memory; if it is determined that the current available resources of the target GPU meet the resources indicated by the application resource description, the resources indicated by the application resource description in the target GPU are allocated to the application, including: if it is determined that the available video memory of the target GPU is greater than or equal to the rated video memory, video memory of the rated video memory size in the target GPU is allocated to the application, and the available video memory of the target GPU is updated.

[0010] In another possible implementation, the above method also includes: obtaining a resource release request from the application, the resource release request including a storage identifier, the storage identifier indicating the video memory used by the application in the target GPU; releasing the video memory indicated by the storage identifier in the target GPU, and updating the available video memory of the target GPU.

[0011] In another possible implementation, the application resource description includes the rated computing power utilization of the application; the resource request is used to request the use of the computing power of the target GPU; during the process of the application using the target GPU resources, the application is controlled to use the target GPU resources less than or equal to the resources indicated by the application resource description, including: during the process of the application using the target GPU resources, obtaining the real-time computing power utilization of the target GPU by the application; when the real-time computing power utilization is greater than the rated computing power utilization, delaying the execution of the application's instruction to use the target GPU computing power until the obtained real-time computing power utilization is less than or equal to the rated computing power utilization.

[0012] In another possible implementation, creating a virtual GPU corresponding to an application includes: hijacking a target GPU's underlying function for calling resources to obtain the virtual GPU.

[0013] In another possible implementation, the computing power includes one or more of the following: computing power, encoding power, and decoding power.

[0014] The method provided in the embodiment of the present application selects a suitable target GPU based on the application's application resource description and controls the application's use of the target GPU's resources. This solution performs control based on the application resource description, ensuring that the application's resource usage of the target GPU is within a reasonable range, or that the application is allowed to use the target GPU's resources only when the target GPU's available resources are sufficient. This allows the application to obtain resource isolation functionality to ensure that when GPU resources are fully utilized in shared mode, the problems of resource preemption and exception propagation that exist when multiple applications share a GPU will not occur, thereby ensuring the stability of GPU operation.

[0015] In a second aspect, the present application provides a GPU resource scheduling device, which includes: an acquisition module and a processing module; the acquisition module is used to obtain an application resource description of an application and determine a target GPU that meets the application resource description; the application resource description is used to indicate the GPU capability requirements of the application; the acquisition module is also used to obtain a start application request for starting the application and start the application; the processing module is used to, when obtaining the resource request of the application, if it is determined that the current available resources of the target GPU meet the resources indicated by the application resource description, allocate the resources indicated by the application resource description in the target GPU to the application; or, after obtaining the resource request of the application, during the process of the application using the target GPU resources, control the application to use the target GPU resources less than or equal to the resources indicated by the application resource description.

[0016] In one possible implementation, the apparatus further includes: a creation module; the creation module is used to create a virtual GPU corresponding to the application; the processing module is specifically used to, upon obtaining a resource request from the application, allocate the resources indicated by the application resource description in the target GPU to the application when the virtual GPU determines that the currently available resources of the target GPU meet the resources indicated by the application resource description; the processing module is specifically used to, after obtaining the resource request from the application, control the virtual GPU to use the target GPU resources so that they are less than or equal to the resources indicated by the application resource description during the process of the application using the target GPU resources.

[0017] In another possible implementation, the application resource description includes the rated video memory of the application; the resource request is used to allocate video memory; and the processing module is specifically used to, if it is determined that the available video memory of the target GPU is greater than or equal to the rated video memory, allocate video memory of the rated video memory size in the target GPU to the application, and update the available video memory of the target GPU.

[0018] In another possible implementation, the acquisition module is further used to obtain a resource release request from the application, where the resource release request includes a storage identifier, where the storage identifier indicates the video memory used by the application in the target GPU; the processing module is further used to release the video memory indicated by the storage identifier in the target GPU and update the available video memory of the target GPU.

[0019] In another possible implementation, the application resource description includes the rated computing power utilization of the application; the resource request is used to request the use of the computing power of the target GPU; the processing module is specifically used to obtain the real-time computing power utilization of the target GPU by the application during the process of the application using the target GPU resources; when the real-time computing power utilization is greater than the rated computing power utilization, delaying the execution of the application's instruction to use the target GPU computing power until the obtained real-time computing power utilization is less than or equal to the rated computing power utilization.

[0020] In another possible implementation, the creation module is specifically configured to hijack the underlying function of the target GPU for calling resources to obtain a virtual GPU.

[0021] In another possible implementation, the computing power includes one or more of the following: computing power, encoding power, and decoding power.

[0022] In a third aspect, the present application provides a server comprising: a processor and a memory; the memory stores instructions executable by the processor; when the processor is configured to execute the instructions, the server implements the method of the first aspect above.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium, which includes: computer software instructions; when the computer software instructions are executed in an electronic device, the electronic device implements the method of the first aspect above.

[0024] In a fifth aspect, the present application provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the steps of the relevant method described in the first aspect to implement the method of the first aspect.

[0025] The beneficial effects of the second to fifth aspects mentioned above can be referred to the corresponding description of the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of the application environment of a GPU resource scheduling method provided in this application;

[0027] Figure 2 A schematic diagram of the architecture of a GPU resource scheduling device provided in this application;

[0028] Figure 3 A schematic diagram of the composition of a local GPU resource management module provided in this application;

[0029] Figure 4 A flowchart of a GPU resource scheduling method provided in this application;

[0030] Figure 5 A flowchart of another GPU resource scheduling method provided in this application;

[0031] Figure 6 A schematic diagram of a process for allocating video memory provided by this application;

[0032] Figure 7 A schematic diagram of a process for releasing video memory provided by this application;

[0033] Figure 8 A flowchart of computing power control provided by this application;

[0034] Figure 9 A schematic diagram of the composition of a GPU resource scheduling device provided in this application;

[0035] Figure 10 A schematic diagram of the composition of a server provided for this application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0037] It should be noted that in the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described in the embodiments of this application as "exemplarily" or "for example" should not be interpreted as being more preferred or advantageous than other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0038] In order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order.

[0039] How to fully utilize the computing power of GPUs to solve complex computational problems and find appropriate GPU resource scheduling solutions have become one of the research hotspots in today's computer technology. Current GPU resource scheduling solutions include exclusive mode. Exclusive mode means that a single application completely occupies a single GPU. In this mode, the GPU's performance cannot be fully utilized, resulting in wasted GPU resources and inefficient utilization. Therefore, GPU resource scheduling solutions also include shared mode. Shared mode means that multiple applications share a single GPU. Although this can to some extent solve the problem of wasted GPU resources and inefficient utilization, this mode is subject to problems such as multiple applications competing for resources, exceptions propagating to each other, and unstable GPU operation.

[0040] To sum up, how to ensure that applications do not preempt resources and that the GPU can run stably while effectively utilizing GPU resources is an urgent problem to be solved.

[0041] Based on this, an embodiment of the present application provides a GPU resource scheduling method, which controls the application's use of GPU resources according to the application resource description, avoids the problem of resource preemption in shared mode, and improves the operating stability of the GPU.

[0042] The GPU resource scheduling method provided in this application can be applied to Figure 1 In the application environment shown. Figure 1 As shown, the application environment may include: a central device 101 and multiple sub-servers 102. The central device 101 and the multiple sub-servers 102 are connected.

[0043] The central device 101 can be implemented in a server. The server referred to herein can be a server cluster consisting of multiple servers, a single server, or a computer. The central device 101 can specifically be a processor or processing chip in the server. The embodiments of this application do not limit the specific device form of the server. Figure 1 In the figure, the central device 101 is applied to a single server as an example.

[0044] The sub-server 102 can be applied to a server, and each sub-server 102 can be provided with one or more GPUs. Multiple sub-servers 102 and the central device 101 constitute a server cluster. Each sub-server 102 can be called a node. When scheduling GPU resources, the user can query the appropriate target node through the central device 101 so that the user can use the GPU resources in the target node. The embodiment of the present application does not limit the specific form and number of sub-servers 102. Figure 1 In the figure, the number of sub-servers 102 is 4 as an example.

[0045] Figure 2 This is a schematic diagram of the architecture of a GPU resource scheduling device (or resource scheduling device) provided in an embodiment of the present application. The resource scheduling device can be deployed in the above Figure 1 Any server shown in the application environment. Figure 2 As shown in the figure, the architecture consists of four parts, namely the application layer, resource layer, virtualization layer and hardware layer from top to bottom.

[0046] Application layer: Users can use machine learning platforms at the application layer (such as TensorFlow and PyTorch applications) to generate application resource descriptions that indicate the application's GPU capability requirements based on their actual needs and the proportions of various resources. This description is then distributed to the resource layer. GPU capability requirements refer to the application's requirements for computing power, video memory, and other capabilities.

[0047] Resource layer: Based on the application resource description, an appropriate GPU is selected from the pre-established GPU resource pool. A virtual GPU (vGPU) is created in the virtualization layer, and the application is launched. A virtual GPU is a hijacked function, essentially adding new functionality to the underlying function. This virtual GPU is used to control the application's use of GPU resources. The specific control process and virtual GPU creation process are described in the following embodiments and are not detailed here.

[0048] Virtualization layer: Runs applications and implements resource isolation between different applications when accessing GPU resources according to the application resource description of the application, that is, applications do not compete for resources with each other.

[0049] Hardware layer: includes GPU hardware and drivers, providing resources for application running.

[0050] Among them, the application layer and resource layer can be integrated into Figure 1 In the central device 101, the virtualization layer and the hardware layer can be integrated into Figure 1 In each sub-server 102.

[0051] The resource layer includes a cluster GPU resource management module and an application management module. The cluster GPU resource management module is responsible for aggregating the GPUs of all nodes and pooling them according to user operations (for example, according to GPU type) (i.e., dividing multiple GPU sets according to user operations); it is responsible for managing, recording, and persisting the resource allocation of all GPUs; it is responsible for determining the available GPUs for use by the application based on the resource conditions of the application resource description of the application, and requesting the creation and deletion of virtual GPUs (i.e., releasing hijacking) from the local GPU resource management module of the node corresponding to the available GPU. The application management module is responsible for the creation, query, modification, deletion, status management, and persistence of application resource descriptions (i.e., transferring application resource descriptions to a database for long-term storage), and scheduling GPU resources based on the GPU resource conditions and GPU priority.

[0052] The virtualization layer includes a native GPU resource management module, a GPU virtualization module, and an application loader.

[0053] The GPU virtualization module is responsible for creating a virtual GPU and providing an operating environment for applications. By hijacking and replacing underlying functional functions, it allows applications to obtain resource isolation capabilities without being aware of the underlying functions. The GPU virtualization module can also be divided into the following three sub-modules: the communication module, the hijacking module, and the control module. The communication module is responsible for requesting the use of GPU resources from the local GPU resource management module. The hijacking module hijacks the underlying functional functions provided by the GPU manufacturer driver through functions to create a virtual GPU. The control module implements control of video memory, computing power, encoding power, and decoding power.

[0054] The local GPU resource management module is responsible for receiving and executing control signals issued by the GPU resource management module; regularly reporting GPU status, availability, etc. to the cluster GPU resource management module for cluster GPU operation and maintenance; and providing necessary information such as real-time GPU utilization (or real-time computing power utilization) to the GPU virtualization module through the communication channel. Figure 3As shown, the local GPU resource management module includes the following four modules: communication module, information collection module, resource management module and interface module (or called rest service module). The communication module is used to communicate with the GPU virtualization module. The communication content includes operation commands (such as the application and release of vGPU) and real-time data communication (such as the computing power utilization of the process, etc.). The information collection module is used to collect GPU information and status and GPU hardware information and status, and send GPU information to other modules (such as communication module, resource management module and interface module). The resource management module receives the configuration from the cluster GPU resource management module and allocates and manages GPU resources according to certain strategies. The interface module provides an external interface, such as connecting to the cluster GPU resource management module, receiving the relevant operation instructions of the vGPU issued and sending them to the resource management module.

[0055] Figure 4 The flowchart of a GPU resource scheduling method provided by the embodiment of the present application is shown in FIG. Figure 1 In the application environment shown, and Figure 2 The system architecture shown.

[0056] The GPU resource scheduling method provided in this application primarily implements centralized scheduling of cluster GPU resources through a cluster GPU resource management module in a centralized server. Therefore, the cluster GPU resource management module pre-collects and integrates resource information from each GPU in all sub-servers (or nodes) connected to the centralized server for subsequent execution of the GPU resource scheduling method.

[0057] As previously mentioned, the local GPU resource management module of each sub-server includes an information collection module for collecting GPU hardware information. The cluster GPU resource management module can obtain GPU resource information from the local GPU resource management module of each node, including GPU number, GPU type, power, driver version, memory size, computing power, and other information. The cluster GPU resource management module can classify GPUs according to specific rules, pooling and categorizing GPUs across all nodes to form different resource pools.

[0058] In the case where the application resource description indicates that the application requires a GPU, for example, the GPUs of the same GPU type in each node can be grouped together. For example, graphics cards of type T4 can be grouped together into a set, such as a T4 resource pool. Graphics cards of type V100 can be grouped together into a set, such as a V100 resource pool (where T4 and V100 are both specific graphics card models), so that it is convenient to subsequently search from the set to the specific graphics card, and from coarse to fine to select the appropriate GPU according to the application resource description. When scheduling queries for available GPUs, there may be multiple GPUs that meet the resource requirements of the application. In this case, it can be determined which GPU to use based on the priority of these GPUs (such as a pre-determined order of use, or sorting by latency, etc.).

[0059] If the application resource description indicates that the application requires two or more GPUs, the application cannot run on multiple hosts simultaneously. Therefore, the multiple GPUs required by the application should be located on the same node. For example, GPUs on the same node can be grouped together, and further, nodes with the same number of GPUs can be grouped together. For example, nodes with one GPU installed can be grouped together, nodes with two GPUs installed can be grouped together, and so on. When performing a target GPU query, one can first query for nodes whose number of GPUs meets the application's requirements, and then further query for multiple GPUs within the node whose resources meet the requirements.

[0060] like Figure 4 As shown, the GPU resource scheduling method provided in this application may specifically include the following steps:

[0061] S401: A resource scheduling apparatus obtains an application resource description of an application program and determines a target GPU that satisfies the application resource description.

[0062] The application resource description is used to indicate the GPU capability requirements of the application.

[0063] In some embodiments, when a user needs to use GPU resources, the resource scheduling device can configure an application resource description indicating the GPU capability requirements of the application program, so that the resource scheduling device can obtain and determine the target GPU that meets the application resource description based on the application resource description.

[0064] In S401, based on the requirements of the application resource description, the target GPU determined may be one GPU or multiple GPUs. When the target GPUs are multiple GPUs, they may be multiple GPUs on the same node.

[0065] Figure 5 Indicates a Figure 2In the schematic architecture, this application provides a process of GPU resource scheduling method. Figure 5 , the specific implementation of S401 is described. As mentioned above, the application management module is responsible for the creation and modification of application resource descriptions. Users can Figure 2 In the application layer, the machine learning platform (e.g., TensorFlow and PyTorch applications) configures the GPU capability requirements of the application to be run, including GPU type, memory size, computing power, etc., obtains an application resource description and sends it to the application management module. The application management module further sends this application resource description to the cluster GPU resource management module. The cluster GPU resource management module then determines the target GPU that meets the application resource description based on the application resource description sent by the application management module.

[0066] The cluster GPU resource management module records the information of each GPU in advance and generates different resource pools according to certain rules. Therefore, taking the case where the aforementioned application requires a GPU, the resource pool is divided according to the GPU type in advance as an example to illustrate the specific process of determining the target GPU. The cluster GPU resource management module can determine the corresponding type of resource pool based on the GPU type in the application resource description. Furthermore, based on the requirements such as the memory size and computing power size in the application resource description, a GPU whose memory and computing power meet the application resource description is selected from the current type of resource pool, and then the target GPU is determined according to the pre-set priority rules. For example, the GPU number that meets the requirements is recorded to mark the target GPU.

[0067] S402: The resource scheduling device obtains an application startup request for starting an application, and starts the application.

[0068] In some embodiments, the resource scheduling apparatus may obtain a start application request for starting an application, and in response to the request, start an application corresponding to the request.

[0069] Combine Figure 5 , describing the specific implementation of S402. The application resource description also includes the application identifier. In the resource scheduling device, the application loader in the target node (i.e., the node where the target GPU is located) can receive a launch application request sent by the application management module. The launch application request includes the application identifier. In response to the request, the application loader injects the application identifier into the GPU virtualization module to launch the application corresponding to the identifier in the GPU virtualization module.

[0070] In other embodiments, before launching an application, the resource scheduling device also creates a virtual GPU corresponding to the application. The number of created virtual GPUs is equal to the number of target GPUs, and the virtual GPUs correspond one-to-one with the target GPUs. Each virtual GPU is used to control access to its corresponding target GPU.

[0071] Exemplarily, the resource scheduling device uses a function hijacking method to hijack the underlying function of the target GPU for calling resources to obtain a virtual GPU. The virtual GPU can control the application to use the resources of the target GPU, as shown in S403.

[0072] Specifically, such as Figure 5 As shown, after the cluster GPU resource management module determines the target GPU, it can generate a virtual GPU identifier (vGPU-ID) and send a creation request to the local GPU resource management module in the target node to request the creation of a virtual GPU. The creation request includes the virtual GPU identifier and the application resource description. Further, in response to the creation request, the local GPU resource management module instructs the GPU virtualization module to create a virtual GPU corresponding to the application resource description. The local GPU resource management module binds and saves the application resource description and the virtual GPU identifier so that the subsequent GPU virtualization module can request the application resource description from the local GPU resource management module based on the virtual GPU identifier to implement subsequent resource usage control.

[0073] It should be noted that if there are multiple target GPUs, the above steps are performed separately for each target GPU, generating a virtual GPU identifier corresponding to the target GPU, and sending a creation request for each virtual GPU to the local GPU resource management module to create multiple virtual GPUs. The local GPU resource management module binds and stores the application resource description and the identifier of each virtual GPU.

[0074] In addition, after determining the target GPU and generating the virtual GPU, the cluster GPU resource management module also sends the target node information and the virtual GPU identifier to the application management module, so that the application management module sends a start application request to the application loader in the target node based on the target node information. The start application request includes the identifier of the application and the identifier of the virtual GPU. The target node information can be the identifier of the target node, so that the application management module can determine the location of the target node. The embodiment of the present application does not limit the specific content of the target node information. In response to the start application request, the application loader is injected into the GPU virtualization module to start the corresponding application in the GPU virtualization module. In addition, the GPU virtualization module can use the identifier of the virtual GPU in the start application request to apply to the local GPU resource management module for the use of the application resource description to control the resource usage of the application according to the application resource description.

[0075] The process of controlling the application program to access the resources of the target GPU corresponding to each virtual GPU according to the application resource description is consistent, as described in S403 below.

[0076] S403. When the resource scheduling device obtains the resource request of the application, if it determines that the current available resources of the target GPU meet the resources indicated by the application resource description, it allocates the resources indicated by the application resource description in the target GPU to the application; or, after obtaining the resource request of the application, during the process of the application using the target GPU resources, it controls the application to use the target GPU resources that are less than or equal to the resources indicated by the application resource description.

[0077] In some embodiments, when a resource scheduling device creates a virtual GPU, upon receiving a resource request from an application, the virtual GPU allocates the resources indicated by the application resource description in the target GPU to the application if it determines that the currently available resources of the target GPU meet the resources indicated by the application resource description.

[0078] Exemplarily, the application resource description includes the application's rated video memory. The resource request is used to allocate video memory. If the virtual GPU determines that the target GPU's available video memory is greater than or equal to the rated video memory, the virtual GPU allocates video memory equal to the rated video memory size of the target GPU to the application, and updates the target GPU's available video memory.

[0079] Specifically, as mentioned above, the virtual GPU uses function hijacking to hijack the target GPU's underlying resource-calling functions. In this example, these underlying functions are provided by the target GPU's manufacturer driver for requesting or releasing video memory. Applications can initiate resource requests by calling these underlying functions.

[0080] However, GPU resources are global public resources, and each application can use underlying functions to request resources based on its needs. If an application has a flaw, such as repeatedly occupying resources and not releasing them, it can lead to unstable supply of public GPU resources, which in turn affects the normal operation of other applications. Therefore, when multiple applications use the same GPU resources, there will be competition for resources among them, leading to unstable GPU operation.

[0081] To address the above issues, the embodiments of this application use a function hijacking method to add a judgment function to the underlying function, allowing applications to obtain resource isolation functions without perception, so as to control the application's use of GPU resources within a reasonable range, ensure that resource preemption issues do not occur, and ensure the stability of GPU operation. Among them, function hijacking refers to hijacking a function before it runs and adding the required functions. However, this method does not affect the effectiveness of the original function and does not affect the basic resource calling function.

[0082] Video memory may be frequently requested and released while an application is running. Therefore, in addition to supporting the recording of the rated number of GPUs used by an application and the rated total resources of each GPU, the GPU virtualization module must also ensure that the application's video memory usage on each GPU does not exceed the total video memory of that GPU at any given time. This embodiment of the application uses a virtual GPU to perform a layer of logical judgment when an application uses video memory, thereby precisely controlling the target GPU's video memory usage.

[0083] Figure 6 A schematic diagram of a process for allocating video memory provided in an embodiment of the present application, combined with Figure 6The above-mentioned process of applying for video memory is explained. The application can call the virtual GPU to request the GPU virtualization module to allocate video memory. After receiving the request, the GPU virtualization module checks whether the available video memory of the target GPU (the GPU virtualization module can obtain and save the available video memory through the local GPU resource management module) meets (is greater than or equal to) the rated video memory in the application resource description. If not, the response code 1 (for example, described as CUDA_ERROR_OUT_OF_MEMORY) is returned to the user side, so that the user can determine the problem based on the response code. If it is satisfied, the GPU virtualization module allocates video memory of the rated video memory size (that is, the rated video memory required by the application in the application resource description). Furthermore, if the allocation of video memory fails, the original error code is returned to the user side, so that the user can determine the reason for the allocation failure based on the error code. If the allocation is successful, the GPU virtualization module updates the available video memory of the target GPU (the available video memory size minus the rated video memory size) and saves a storage identifier (e.g., a pointer) indicating the video memory used by the current application in the target GPU (or the video memory allocated to the application), and returns a response code 2 (e.g., described as CUDA_SUCCESS), indicating that the user has successfully allocated the video memory.

[0084] In addition, the application can also release the video memory after using it. Specifically:

[0085] 1. Obtain a resource release request from the application. The resource release request includes a storage identifier, which indicates the video memory used by the application in the target GPU.

[0086] 2. Release the video memory indicated by the storage identifier in the target GPU and update the available video memory of the target GPU.

[0087] Figure 7 A schematic diagram of a process for releasing video memory provided in an embodiment of the present application, combined with Figure 7 The above-mentioned process of releasing video memory is explained. The application sends a resource release request to the GPU virtualization module by calling the virtual GPU. The resource release request carries the above-mentioned storage identifier (such as a pointer). After receiving the resource release request, the GPU virtualization module queries whether the pointer exists. If it does not exist, it returns a response code 3 (for example, described as CUDA_ERROR_INVALID_VALUE) to the user side. If it exists, the GPU virtualization module releases the video memory indicated by the pointer in the target GPU. Furthermore, if the release of video memory fails, the original error code is returned to the user side. If the video memory is released successfully, the GPU virtualization module updates the available video memory of the target GPU (the available video memory size of the target GPU plus the rated video memory size), removes the storage record of the pointer, and returns a response code 2 to indicate to the user that the video memory release is successful.

[0088] In other embodiments, when the resource scheduling device creates a virtual GPU, after obtaining the resource request of the application, during the process of the application using the target GPU resources, the virtual GPU controls the application to use the target GPU resources less than or equal to the resources indicated by the application resource description.

[0089] Exemplarily, the application resource description includes the application's rated computing power utilization. The resource request is used to request the use of the target GPU's computing power. After obtaining the application's request for GPU computing power, the application's real-time computing power utilization of the target GPU is obtained while the application is using the target GPU resources. If the real-time computing power utilization is greater than the rated computing power utilization, execution of the application's instruction to use the target GPU's computing power is delayed until the obtained real-time computing power utilization is less than or equal to the rated computing power utilization.

[0090] It should be noted that computing power includes computing power, encoding power and decoding power. Among them, encoding and decoding power can be used in the process of encoding and decoding video information. The control methods of the above three computing power resources are the same, the difference lies in the different underlying function functions called. The control of computing power resources mainly lies in ensuring that the real-time computing power utilization of the process running the application is stable within the expected range (that is, the real-time computing power utilization is less than or equal to the rated value) when the application calls the underlying function function to use computing power. The control method is also achieved by hijacking the underlying function function called by the computing power to obtain a virtual GPU, which will not be repeated here.

[0091] Figure 8 A flowchart of a computing power control method provided in this application embodiment, combined with Figure 8The above-mentioned computing resource control process is described below. After the GPU virtualization module requests use of the virtual GPU and receives the application resource description issued by the local GPU resource management module, the application calls the virtual GPU to use the computing resources of the target GPU. During the use of the target GPU resources, the local GPU resource management module periodically obtains the real-time computing utilization of the target GPU by the application process and updates it to shared memory. The GPU virtualization module can periodically obtain the real-time computing utilization from the shared content and compare it with the rated computing utilization in the application resource description using the judgment function provided by the virtual GPU. If the real-time computing utilization is greater than the rated computing utilization, it indicates that the application is currently consuming too much computing resources. The GPU virtualization module can delay the execution of the application's instructions to use the target GPU computing resources until the obtained real-time computing utilization is less than or equal to the rated computing utilization. The specific delay process can adopt a gradually increasing or decreasing delay granularity method, adding a delay instruction to the application's instructions to use computing resources. The delay size can be continuously optimized and adjusted based on the rate of decline of real-time computing utilization. Simply put, the faster the real-time computing utilization decreases, the smaller the delay size can be set. If real-time computing power utilization decreases slowly, the latency can be set higher. By using this approach, the real-time computing power utilization of the application process can be controlled to be less than or equal to the rated computing power utilization specified in the application resource description. This allows the application to use computing power resources within the expected range, preventing resource preemption and unstable GPU operation.

[0092] It should be noted that the resource control process of S403 above is described by taking the example of an application using one GPU resource (i.e., the target GPU that satisfies the application resource description of the application is one). If the application resource description instructs the application to use multiple GPUs (i.e., the target GPU that satisfies the application resource description of the application is multiple), multiple virtual GPUs are created, and the control process of each virtual GPU for the application to use resources is consistent with the method described in S403, and will not be repeated here. For example, if the application resource description instructs the application to use two GPUs, the resource requirements of each GPU are indicated in the application resource description. If the application resource description indicates that the rated computing power utilization of one GPU is 30% and the rated computing power utilization of another GPU is 20%, two virtual GPUs are generated, respectively attached to the two GPUs whose resources are satisfied by the scheduling query, and the application uses the resources of the two GPUs by calling these two virtual GPUs. Each virtual GPU is independently controlled according to its own part of the resource description.

[0093] The following combination Figure 2The system architecture shown here comprehensively illustrates the complete process of this solution. 1. The cluster GPU resource management module obtains GPU resource information from each node's local GPU resource management module. Based on certain rules (for example, grouping GPUs of the same type together), all nodes' GPUs are pooled by type, resulting in resource pools of different GPU types. 2. The user configures a resource description through the machine learning platform and sends it to the application management module. 3. The application management module requests an available GPU from the cluster GPU resource management module. The cluster GPU resource management module searches for a suitable resource pool, selects the target GPU, and sends a create virtual GPU instruction to the local GPU management module of the node where the target GPU resides. The cluster GPU resource management module then returns the virtual GPU identifier and target node information to the application management module. 4. The local GPU management module binds the virtual GPU identifier and application resource description and creates a virtual GPU in the GPU virtualization module. 5. The application management module sends an application launch request to the target node's application loader. The application loader injects the GPU virtualization module and launches the corresponding application. 6. After the application is launched, the GPU virtualization module uses the virtual GPU identifier to request an application resource description from the local GPU management module. After confirmation, the local GPU management module sends the resource description to the GPU virtualization module. The GPU virtualization module isolates the resources according to the application resource description (ie, the above-mentioned S403).

[0094] The technical solutions provided by the above embodiments bring at least the following beneficial effects: the GPU resource scheduling method provided by the embodiments of the present application selects a suitable target GPU according to the application resource description of the application, and controls the application's use of the target GPU's resources. This solution is controlled based on the application resource description, so that the application's resource usage of the target GPU is within a reasonable range, or the application is allowed to use the target GPU's resources only when the target GPU's available resources are sufficient. The application is given the function of resource isolation to ensure that when the GPU resources are fully utilized in the shared mode, there will be no problems of resource preemption and abnormal propagation that exist when multiple applications share a GPU, thereby ensuring the stability of the GPU operation.

[0095] Furthermore, this solution uses a function hijacking underlying function to create a virtual GPU. Through the virtual GPU, the application can precisely control the amount of GPU resources used to achieve resource isolation, effectively solving the problem of resource preemption in the shared mode. In addition, this solution integrates video memory, computing power, encoding power, and decoding power for unified scheduling in the cluster. It can configure the appropriate GPU according to user needs, solve the resource waste problem that exists in the scheduling of the entire GPU card, and effectively improve the resource utilization of the cluster GPU. In addition, users can define appropriate resource quotas by configuring application resource descriptions to meet the GPU usage requirements in various scenarios.

[0096] In an exemplary embodiment, the present application also provides a resource scheduling device. The resource scheduling device may include one or more functional modules for implementing the GPU resource scheduling method of the above method embodiment. For example, Figure 9 This is a schematic diagram of the composition of a resource scheduling device provided in an embodiment of the present application. Figure 9 As shown, the device includes: an acquisition module 901 and a processing module 902. The acquisition module 901 and the processing module 902 are connected to each other.

[0097] The acquisition module 901 is used to acquire an application resource description of the application and determine a target GPU that meets the application resource description; the application resource description is used to indicate the GPU capability requirements of the application.

[0098] The acquisition module 901 is further configured to acquire an application startup request for starting an application and start the application.

[0099] Processing module 902 is configured to, upon receiving a resource request from an application, allocate the resources indicated by the application resource description in the target GPU to the application if it is determined that the currently available resources of the target GPU meet the resources indicated by the application resource description. Alternatively, processing module 902 is configured to, after receiving a resource request from an application, control the application to use target GPU resources that are less than or equal to the resources indicated by the application resource description while the application is using the target GPU resources.

[0100] In some embodiments, the above device further includes: a creation module 903.

[0101] The creation module 903 is used to create a virtual GPU corresponding to the application.

[0102] The processing module 902 is specifically configured to, upon receiving a resource request from an application, allocate the resources indicated by the application resource description in the target GPU to the application if the virtual GPU determines that the currently available resources of the target GPU meet the resources indicated by the application resource description.

[0103] The processing module 902 is specifically configured to, after obtaining a resource request from the application, control the application to use target GPU resources so that the target GPU resources are less than or equal to the resources indicated by the application resource description during the application's use of target GPU resources.

[0104] In some embodiments, the application resource description includes the rated video memory of the application; the resource request is used to allocate video memory; the processing module 902 is specifically used to allocate video memory of the rated video memory size in the target GPU to the application if it is determined that the available video memory of the target GPU is greater than or equal to the rated video memory, and update the available video memory of the target GPU.

[0105] In some embodiments, the acquisition module 901 is further configured to acquire a resource release request from the application, where the resource release request includes a storage identifier, and the storage identifier indicates a video memory in the target GPU used by the application.

[0106] The processing module 902 is further configured to release the video memory indicated by the storage identifier in the target GPU and update the available video memory of the target GPU.

[0107] In some embodiments, the application resource description includes the rated computing power utilization of the application; the resource request is used to request the use of the computing power of the target GPU; the processing module 902 is specifically used to obtain the real-time computing power utilization of the target GPU by the application during the process of the application using the target GPU resources; when the real-time computing power utilization is greater than the rated computing power utilization, delaying the execution of the application's instruction to use the target GPU computing power until the obtained real-time computing power utilization is less than the rated computing power utilization.

[0108] In some embodiments, the creation module 903 is specifically configured to hijack the underlying functional function of the target GPU for calling resources to obtain a virtual GPU.

[0109] In some embodiments, computing power includes one or more of the following: computing power, encoding power, and decoding power.

[0110] In an exemplary embodiment, the present application further provides a server, which may be the resource scheduling device in the above method embodiment. Figure 10 This is a schematic diagram of the composition of the server provided in the embodiment of this application. Figure 10 As shown, the server may include: a processor 1001 and a memory 1002; the memory 1002 stores instructions executable by the processor 1001; when the processor 1001 is configured to execute the instructions, the server implements the method described in the aforementioned method embodiment.

[0111] In an exemplary embodiment, the present application also provides a computer-readable storage medium having computer program instructions stored thereon; when the computer program instructions are executed by a computer, the computer implements the method described in the aforementioned embodiment. The computer may be the aforementioned server. The computer-readable storage medium may be a non-transitory computer-readable storage medium, for example, a non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0112] In an exemplary embodiment, the present application also provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related method steps to implement the GPU resource scheduling method in the above-mentioned embodiment.

[0113] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple components. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0114] Although the present application has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely illustrative of the present application as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, the present application is intended to include such modifications and variations as fall within the scope of the claims of the present application and their equivalents.

[0115] The above are only specific embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for scheduling resources of a graphics processing unit (GPU), characterized in that: The method comprises: Obtaining an application resource description of the application, and determining a target GPU that satisfies the application resource description; the application resource description is used to indicate the GPU capability requirements of the application; the application resource description includes a rated computing power utilization rate of the application; Obtaining a start application request for starting the application, and starting the application; Upon obtaining the resource request of the application, if it is determined that currently available resources of the target GPU satisfy the resources indicated by the application resource description, after obtaining the resource request of the application, during the process of the application using the resources of the target GPU, controlling the application to use resources of the target GPU that are less than or equal to the resources indicated by the application resource description; the resource request is used to request the use of computing power of the target GPU; The step of controlling the application to use the target GPU resources so that the target GPU resources are less than or equal to the resources indicated by the application resource description during the application using the target GPU resources includes: During the process of the application using the target GPU resources, obtaining the real-time computing power utilization rate of the target GPU by the application; When the real-time computing power utilization is greater than the rated computing power utilization, the execution of the instruction of the application program to use the target GPU computing power is delayed until the acquired real-time computing power utilization is less than or equal to the rated computing power utilization.

2. The method according to claim 1, characterized in that Before starting the application, the method further includes: Creating a virtual GPU corresponding to the application; After obtaining the resource request of the application, in the process of the application using the target GPU resource, controlling the application to use the target GPU resource that is less than or equal to the resource indicated by the application resource description includes: After obtaining the resource request of the application, during the process of the application using the target GPU resources, the virtual GPU controls the application to use the target GPU resources that are less than or equal to the resources indicated by the application resource description.

3. The method according to claim 2, characterized in that The creating a virtual GPU corresponding to the application includes: The function hijacks the underlying function of the target GPU for calling resources to obtain the virtual GPU.

4. The method according to claim 1, wherein The computing power includes one or more of the following: computing power, encoding power, and decoding power.

5. A GPU resource scheduling device, characterized in that: The device comprises: an acquisition module and a processing module; The acquisition module is used to obtain an application resource description of the application and determine a target GPU that meets the application resource description; the application resource description is used to indicate the GPU capability requirements of the application; the application resource description includes the rated computing power utilization of the application; The acquisition module is further configured to acquire an application startup request for starting the application and start the application; The processing module is configured to, upon obtaining a resource request from the application, control the application to use resources of the target GPU that are less than or equal to the resources indicated by the application resource description during the process of the application using the target GPU resources if it is determined that the currently available resources of the target GPU meet the resources indicated by the application resource description after obtaining the resource request from the application; the resource request is used to request the use of computing power of the target GPU; The processing module is specifically used to obtain the real-time computing power utilization of the target GPU by the application during the process of the application using the target GPU resources; when the real-time computing power utilization is greater than the rated computing power utilization, delay the execution of the instruction of the application to use the computing power of the target GPU until the obtained real-time computing power utilization is less than or equal to the rated computing power utilization.

6. The device according to claim 5, characterized in that The apparatus further comprises: a creation module; The creation module is used to create a virtual GPU corresponding to the application; The processing module is specifically configured to, after obtaining the resource request of the application, control the application to use the target GPU resources so that the target GPU resources are less than or equal to the resources indicated by the application resource description during the process of the application using the target GPU resources; The creation module is specifically configured to hijack the underlying function of the target GPU for calling resources to obtain the virtual GPU; The computing power includes one or more of the following: computing power, encoding power, and decoding power.

7. A server, characterized in that: The server includes: a processor and a memory; The memory stores instructions executable by the processor; When the processor is configured to execute the instructions, the server implements the method according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium includes: computer software instructions; When the computer software instructions are executed in a computer, the computer is enabled to implement the method according to any one of claims 1 to 4.

Images