GPU task queue management method, system, and device

Abstract

The present invention discloses a GPU task queue management method, system, and device. The method comprises: dividing physical GPU graphics card resources into multiple GPU computing units according to GPU partitioning rules; constructing a GPU resource pool based on the GPU computing units; monitoring the operating status information of the GPU resource pool in real time, obtaining task parameters in the GPU resource pool according to the operating status information, and constructing a task queue with priority according to the task parameters; dynamically allocating to-be-scheduled tasks in the GPU task queue and the GPU computing units based on a resource scheduling strategy, and performing resource scheduling management on the GPU task queue based on the resource parameter information of the GPU computing units. The present invention avoids waste and idleness of resources, improves the management and scheduling efficiency of GPU resources, and realizes the maximum value utilization of the GPU resource pool.

Description

Technical Field

[0001] The present invention relates to the field of computer software technology, and in particular to a GPU task queue management method, system, and device. Background Art

[0002] The next generation of artificial intelligence (AI) technologies based on large language models is developing rapidly. Training, fine-tuning, and deploying large language models requires a significant amount of graphics processing unit (GPU) computing power. Compared to CPU resources, GPU computing power is more expensive and scarce. In large-scale computing environments such as data centers, efficiently managing and scheduling limited GPU resources has become a pressing issue. Current GPU resource management methods have the following main issues:

[0003] Uneven resource allocation: Traditional resource management methods often use static allocation or manual scheduling, which is difficult to adapt to the dynamic changes in task requirements, resulting in uneven resource allocation. Some tasks are waiting due to insufficient resources, while other resources may be idle.

[0004] Low resource utilization: Due to the lack of an effective resource scheduling strategy, GPU resource utilization is generally low. Especially when the task load is uneven, some GPUs are idle for a long time, resulting in resource waste.

[0005] Low task scheduling efficiency: Existing task scheduling mechanisms are often based on simple priority or first-come, first-served principles, lacking comprehensive consideration of task characteristics and resource requirements. This results in low scheduling efficiency and makes it difficult to meet the needs of high-performance computing.

[0006] Lack of automatic scaling capabilities: Existing GPU resource management methods lack the ability to automatically scale out tasks of varying scales and characteristics. They are unable to dynamically adjust resource allocation based on the actual needs of the task, impacting task execution efficiency and the rational use of resources.

[0007] In order to solve the above problems and improve the management and scheduling efficiency of GPU resources, there is an urgent need for a new GPU task queue implementation mechanism that can achieve automatic expansion and contraction of resources, efficient scheduling, and adapt to the needs of different tasks. Summary of the Invention

[0008] The present invention provides a GPU task queue management method, system, and device to improve the management and scheduling efficiency of GPU resources and maximize the value utilization of the GPU resource pool.

[0009] In a first aspect, the present invention discloses a GPU task queue management method, the method comprising:

[0010] According to the GPU division rules, the physical GPU graphics card resources are divided into multiple GPU computing units;

[0011] Building a GPU resource pool based on the GPU computing power unit;

[0012] Monitor the running status information of the GPU resource pool in real time, obtain task parameters in the GPU resource pool according to the running status information, and build a task queue with priority according to the task parameters;

[0013] Based on the resource scheduling strategy, the tasks to be scheduled in the GPU task queue and the GPU computing power unit are dynamically allocated, and the resource scheduling management of the GPU task queue is performed based on the resource parameter information of the GPU computing power unit.

[0014] In some embodiments, the GPU computing power units are distinguished based on resource parameter information, where the resource parameter information includes the manufacturer name of the GPU graphics card, the GPU graphics card type, the unit type, the GPU computing power quantity and / or the GPU memory quantity.

[0015] In some implementations, the resource scheduling strategy includes:

[0016] Determine the optimal execution task and the optimal GPU computing unit based on the task parameters of the task to be scheduled and the resources of the GPU computing unit, and perform task allocation; the task parameters include task priority and the shortest task execution time and waiting time;

[0017] When a new task is received, the new task is inserted into any task queue to be scheduled and queued for scheduling. The GPU computing power unit in the current GPU resource pool is periodically checked to see whether there are idle resources available. The optimal execution task and the optimal GPU computing power unit are determined based on the task parameters of the task to be scheduled and the resource parameter information of the GPU computing power unit.

[0018] In some implementations, performing resource scheduling management on the GPU task queue based on the resource parameter information includes:

[0019] Traverse the currently executing task queue, update the task status that meets the end conditions, and release the corresponding GPU resources to the GPU resource pool;

[0020] Traverse the GPU resource pool, match the idle GPU resources according to the task parameters of the task to be scheduled, allocate the corresponding GPU computing power unit, and start the corresponding task process;

[0021] If there are still tasks to be scheduled in the task queue, determine the tasks that meet the end conditions, update the status of the tasks that meet the end conditions, and release the corresponding GPU resources to the GPU resource pool;

[0022] Allocate the corresponding resources to the task to be scheduled, and put the task that meets the end conditions into the queue of tasks to be scheduled.

[0023] In some embodiments, the GPU computing power unit includes hard partitioning based on MIG and VTPU ​​technologies, soft partitioning based on CUDA, RoCm, and oneAPI programming frameworks, and resource-sharing computing power units.

[0024] In some embodiments, determining the optimal execution task and the optimal GPU computing unit based on the task parameters of the task to be scheduled and the resources of the GPU computing unit further includes:

[0025] Determine the task priority, minimum execution time, waiting time, number of required computing units, and elastic scaling type of the task to be scheduled, and determine the matching idle GPU computing units;

[0026] When there are multiple tasks to be scheduled that match the same idle GPU computing unit, the optimal execution task of the optimal GPU computing unit is determined based on the matching distance and priority.

[0027] In some embodiments, the matching distance is

[0028] gpu_distance=(request_core-target_core)*10+(request_memory-target_memory)

[0029] Among them, gpu_distance is the matching distance, request_core is the amount of GPU computing power required for the task to be scheduled, target_core is the amount of idle GPU computing power, request_memory is the memory capacity of the task to be scheduled, and target_core is the idle GPU memory capacity.

[0030] In some implementations, the termination condition is that the task has been completed or the actual running time of the task exceeds a preset running time.

[0031] In a second aspect, the present invention discloses a GPU task queue management system, comprising:

[0032] A GPU resource pool construction module is used to divide the physical GPU graphics card resources into multiple GPU computing units according to the GPU division rules; and to build a GPU resource pool based on the GPU computing units;

[0033] A GPU task queue construction module is used to monitor the operating status information of the GPU resource pool in real time, obtain task parameters in the GPU resource pool according to the operating status information, and construct a GPU task queue with priority according to the task parameters;

[0034] The GPU task scheduling module dynamically allocates the tasks to be scheduled in the GPU task queue and the GPU computing power unit based on the resource scheduling strategy, and performs resource scheduling management on the GPU task queue based on the resource parameter information of the GPU computing power unit.

[0035] In a third aspect, the present invention discloses a GPU task queue management device, the device comprising:

[0036] a memory storing executable program code;

[0037] a processor coupled to the memory;

[0038] The processor calls the executable program code stored in the memory to execute a GPU task queue management method as described in any one of the above items.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention provides a GPU task queue management method, system, and device. By managing the GPU queue through resource scheduling strategies in combination with task priorities, GPU resources can be dynamically allocated according to the actual needs of the tasks, ensuring that critical tasks can obtain the required resources in a timely manner, thereby improving overall scheduling efficiency, avoiding resource waste and idleness, improving GPU resource management and scheduling efficiency, and realizing maximum value utilization of the GPU resource pool. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of the process provided by the present invention;

[0042] Figure 2 This is a schematic diagram of the structure of the GPU resource pool provided by the present invention;

[0043] Figure 3 This is a schematic diagram of the structure of the GPU task scheduling module provided by the present invention;

[0044] Figure 4 This is a schematic diagram of the relationship between the GPU Driver plugin, GPU Task Operator, and Kubernetes components provided by the present invention;

[0045] Figure 5 This is a flow chart of the GPU task scheduling process provided by the present invention. DETAILED DESCRIPTION

[0046] For better understanding and implementation, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0047] The terms "including" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products or apparatus.

[0048] The embodiment of the present invention discloses a GPU task queue management method, which improves the management and scheduling efficiency of GPU resources and realizes the maximum value utilization of the GPU resource pool.

[0049] Specifically, this method includes the following steps:

[0050] Step S1: Divide the physical GPU graphics card resources into multiple GPU computing power units according to the GPU division rule.

[0051] Traditionally, the industry has generally used a complete graphics card as a scheduling unit. This means that during task scheduling, only one complete graphics card is scheduled, with each graphics card performing its corresponding task. This scheduling method generally results in low GPU resource utilization, especially when task loads are uneven, where some GPUs may be idle for long periods of time, resulting in wasted resources.

[0052] Therefore, in this application, according to the GPU division rules, the physical GPU graphics card resources are divided into multiple GPU computing units of different specifications, and a schedulable GPU resource pool is formed based on these computing units.

[0053] Specifically, the GPU computing power unit can be implemented through GPU hard slicing technology (Nvidia MIG, AMD MxGPU), or by intercepting the GPU computing power and memory call API at the software level, such as interception based on the kernel graphics card driver level, interception based on CUDA driver and SDK, etc.

[0054] The GPU computing units divided in the above manner are all considered basic computing units, and are uniformly managed into the GPU resource pool by distinguishing categories and quantities. It should be noted that GPU in this application also includes NPU and TPU devices, such as Huawei Ascend NPU and Google TPU.

[0055] In this application, GPU task queue management is scheduled at the granularity of GPU computing units. Different types of GPU computing units are suitable for different types of tasks. CPU computing units are named in a unified and standardized manner, which can greatly simplify the matching algorithm between tasks and resources and reduce the complexity of flexible GPU resource scheduling.

[0056] GPU computing units are distinguished based on resource parameter information, which includes the GPU graphics card manufacturer name, GPU graphics card type, unit type, GPU computing power quantity, and / or GPU video memory quantity. For example, the resource name of a GPU computing unit divided into MIG slicing technology for Nvidia's A100 graphics card can be nvidia / a100.mig.4c.20gb, where 4c indicates that the computing unit has 4 GPU computing cores, and 20gb indicates that the computing unit has 20gb of available video memory; the resource name of a GPU computing unit is huawei / ascend910.vnpu.4c.20gb, which indicates that it is a computing unit divided into VNPU slicing technology for Huawei's Ascend 910NPU card. Similarly, a GPU computing unit with a resource name of soft.4c.10gb indicates a 4-core 10GB GPU computing unit implemented through a graphics driver or CUDA call interception. A named shared.10c.80gb indicates a GPU computing unit with shared computing power and memory. Tasks scheduled to this computing unit share the computing power and memory of the same unit.

[0057] Furthermore, GPU computing units can automatically and elastically scale up and down. This scaling includes both the specifications of the GPU computing units and the number of GPU computing units. Furthermore, GPU tasks must also set important parameters such as their priority level, the minimum duration of GPU resource usage per scheduled run, and the maximum cumulative duration of GPU resource usage.

[0058] Step S2: Building a GPU resource pool based on the GPU computing power unit;

[0059] The GPU resource pool has a two-level structure. The first level is the server nodes that contain GPUs in the cluster. The second level is the various GPU computing units owned by each node. Taking a 3-node cluster as an example, the corresponding GPU resource pool structure is as follows: Figure 2 shown.

[0060] Step S3: monitor the operating status information of the GPU resource pool in real time, obtain task parameters in the GPU resource pool according to the operating status information, and construct a task queue with priority according to the task parameters.

[0061] Tasks include GPU tasks in different states, including tasks to be scheduled, tasks in progress, and tasks that have ended. For tasks that require GPU resources, the number of GPU computing units required will be displayed in the task parameters. The GPU resource pool will provide all currently supported types of GPU computing units for selection by tasks to be scheduled. Based on the task parameter information and the resource parameter information of the GPU computing unit, the most suitable GPU computing unit and the corresponding number of computing units are selected. In addition, all tasks also need to set their own priority, the shortest duration of GPU resource occupation for each scheduled run, the maximum cumulative duration of GPU resource occupation, and other task parameters.

[0062] When applying for resources, a GPU task needs to specify the name of the required GPU computing unit and the corresponding number of computing units.

[0063] For example, to apply for two GPU units of nvidia / a100.mig.4c.20gb specification:

[0064] gpu-class=nvidia / a100.mig.4c.20gb

[0065] gpu-count=2

[0066] At the same time, you also need to specify the following three parameters related to task scheduling:

[0067] task-priorty=100, priority. The larger the value, the higher the priority.

[0068] task-one-run-period=3600, the minimum duration of GPU resources occupied by each scheduled run, in seconds

[0069] task-one-run-total=360000, the maximum cumulative time for GPU resource occupation. If this time is exceeded, the task needs to be terminated.

[0070] In this application, the GPU task queue includes the task queue to be scheduled and the task queue being executed. The user creates a GPU task through the interactive interface or API call and submits it to the GPU task scheduling module. The structure of the GPU task scheduling module Figure 3As shown, after a task receiver receives a GPU task, it enters the pending task queue of the GPU task scheduling module and waits for scheduling. The currently executing task queue also has a scheduling daemon thread to implement the specific task scheduling logic. The scheduling daemon thread manages the pending task queue, the running task queue, and the GPU resource pool.

[0071] Step S4: dynamically allocate the tasks to be scheduled in the GPU task queue and the GPU computing power unit based on the resource scheduling strategy, and perform resource scheduling management on the GPU task queue based on the resource parameter information of the GPU computing power unit.

[0072] The scheduling daemon thread of the GPU task scheduling module will dynamically allocate tasks to be scheduled in the GPU task queue and the GPU computing power unit to improve the efficiency of GPU resources. The resource scheduling strategy includes:

[0073] Determine the optimal execution task and the optimal GPU computing unit based on the task parameters of the task to be scheduled and the resources of the GPU computing unit, and perform task allocation; the task parameters include task priority and the shortest task execution time and waiting time;

[0074] When a new task is received, the new task is inserted into any task queue to be scheduled and queued for scheduling. The GPU computing power unit in the current GPU resource pool is periodically checked to see whether there are idle resources available. The optimal execution task and the optimal GPU computing power unit are determined based on the task parameters of the task to be scheduled and the resource parameter information of the GPU computing power unit.

[0075] In the above process, the optimal execution task and the optimal GPU computing unit are determined based on the task parameters of the task to be scheduled and the resources of the GPU computing unit, which also includes:

[0076] Determine the task priority, minimum execution time, waiting time, number of required computing units, and elastic scaling type of the task to be scheduled, and determine the matching idle GPU computing units;

[0077] When there are multiple tasks to be scheduled that match the same idle GPU computing unit, the optimal execution task of the optimal GPU computing unit is determined based on the matching distance and priority.

[0078] Performing resource scheduling management on the GPU task queue based on the resource parameter information includes:

[0079] Traverse the currently executing task queue, update the task status that meets the end conditions, and release the corresponding GPU resources to the GPU resource pool;

[0080] Traverse the GPU resource pool, match the idle GPU resources according to the task parameters of the task to be scheduled, allocate the corresponding GPU computing power unit, and start the corresponding task process;

[0081] If there are still tasks to be scheduled in the task queue, determine the tasks that meet the end conditions, update the status of the tasks that meet the end conditions, and release the corresponding GPU resources to the GPU resource pool;

[0082] Allocate the corresponding resources to the task to be scheduled, and put the task that meets the end conditions into the queue of tasks to be scheduled.

[0083] The specific logic of the GPU task scheduling module is explained below with reference to specific embodiments:

[0084] 1. Traverse the currently running task queue. For tasks that meet the termination conditions, that is, completed tasks or tasks that have timed out (the cumulative running time exceeds task-one-run-total), set the task status to ended and release the corresponding GPU resources to the resource pool;

[0085] 2. Traverse the GPU resource pool, find the scheduled tasks that match the idle GPU resources, allocate the corresponding GPU computing units to them, and start the corresponding task process.

[0086] 3. If there are still tasks to be scheduled in the current task queue, search for task A whose actual running time is greater than task-one-run-period in the currently running task queue. If the GPU resources occupied by task A meet the requirements of a task B to be scheduled, forcibly terminate task A, update the task status of task A, allocate the corresponding resources to task B, start the corresponding task process, and put task A into the task queue to be scheduled for the next round of scheduling.

[0087] In the above task scheduling process, the focus is on how to determine the optimal execution task and the optimal GPU computing unit based on the task parameters of the task to be scheduled and the resources of the GPU computing unit.

[0088] Assume that the currently available GPU resources are two GPU computing units with specifications of huawei / ascend910.vnpu.4c.20gb on Node A.

[0089] The tasks to be scheduled for huawei / ascend910.vnpu are as follows:

[0090] Task 1 requires three huawei / ascend910.vnpu.4c.20gb disks, which can be elastically scaled. Task 2 requires one huawei / ascend910.vnpu.4c.20gb disk, which can be elastically scaled. Task 3 requires two huawei / ascend910.vnpu.4c.30gb disks, which can be elastically scaled. Task 4 requires two huawei / ascend910.vnpu.5c.20gb disks, which can be elastically scaled.

[0091] Task 1 requires three computing units, which doesn't meet Node A's requirements and can't be scheduled on that GPU. However, since tasks 2, 3, and 4 can all be elastically scaled, they meet Node A's requirements. Since the number of GPU computing units required by tasks 3 and 4 is closest to the number of available computing units, tasks 3 and 4 are prioritized. Based on this, the following formula is used to calculate their respective matching distances:

[0092] The matching distance is

[0093] gpu_distance=(request_core-target_core)*10+(request_memory-target_memory)

[0094] Among them, gpu_distance is the matching distance, request_core is the amount of GPU computing power required for the task to be scheduled, target_core is the amount of idle GPU computing power, request_memory is the memory capacity of the task to be scheduled, and target_core is the idle GPU memory capacity.

[0095] According to the above matching distance calculation formula, the calculation results are as follows:

[0096] task3_gpu_distance=(4-4)*10+(30-20)=20

[0097] task4_gpu_distance=(5-4)*10+(20-20)=50

[0098] Therefore, task 3 is scheduled first.

[0099] Furthermore, in some cases, if task 3 or task 4 has the same matching distance, the following logic is used for scheduling:

[0100] 1. Prioritize the tasks that have not been scheduled in the previous task scheduling cycle and have been waiting the longest;

[0101] 2. Priority scheduling has high priority;

[0102] That is, when task 3 was not scheduled in the last task scheduling cycle, and task 4 did not have the last task cycle, or it had the last task cycle but the waiting time did not exceed that of task 3, even if task 4 has a higher priority, task 3 will still be scheduled first.

[0103] When both task 3 and task 4 have not been scheduled in the previous task scheduling cycle and have the same waiting time, and task 4 has a higher priority, task 4 will be scheduled first.

[0104] The above two logics ensure that no low-priority tasks are always in the waiting state. If they have not been scheduled in the previous scheduling cycle, they will be scheduled first in the current scheduling cycle. In addition, it also ensures that high-priority tasks are scheduled first in the same scheduling cycle.

[0105] Furthermore, considering that most current digital transformation applications use cloud-native architecture platforms, and mainstream cloud-native architecture platforms are implemented based on Kubernetes, this application takes Kubernetes as an example to illustrate the specific implementation process and details to pay attention to.

[0106] First, we implement a specific Kubernetes GPU Driver plugin. This plugin runs on every Kubernetes node with GPU resources, divides the GPUs on the node into GPU computing units, and injects them into the Kubernetes API server.

[0107] As a standard kubernetes container resource, the GPU computing unit is allocated and scheduled by kubernetes. At the same time, the allocated resources and remaining resources can also be queried through the kubernetes API, and the GPU task scheduling module is implemented with the kubernetes Operator mechanism. It is specifically implemented as the GPU Task Operator, a mechanism for extending the Kubernetes API that allows developers to use custom resources to manage more complex applications. An Operator usually consists of one or more custom resource definitions (CRDs) and an application running in a Kubernetes cluster. This application is responsible for implementing the logic of custom resources. The relationship between the GPU Driver plugin, GPU Task Operator and kubernetes components is as follows. Figure 4 shown.

[0108] After the user writes a YAML file that defines a GPU task (GpuJob), it is submitted to the API Server through the Kubernetes command line tool or API. The GPU Task Operator monitors the generation of a new task to be scheduled in real time, parses the properties and task parameters of the GPU task, generates the corresponding Task, and puts it into the task queue to be scheduled. When the Task is successfully scheduled, the GPU Task Operator will generate the corresponding Pod instance and write the corresponding GPU resource information to the Pod's resource application attribute request. Subsequently, this Pod instance is scheduled by Kubernetes to a corresponding node for execution. The whole process is as follows: Figure 5 shown.

[0109] The GpuJob CRD object describing the GPU task is modeled as a GpuTask object when it enters the GPU Task Operator. The GpuTask contains some status data related to task scheduling. Some of this status data comes from its corresponding Pod instance, such as whether the Pod is in the pending, running, succeeded, or failed state. The GPU Task Operator automatically synchronizes the GpuTask with the corresponding Pod instance status to complete the resource release and rescheduling logic.

[0110] Furthermore, the present application can also be implemented directly in a Docker container system, which is not limited here.

[0111] Based on the same inventive concept, the present application also provides a GPU task queue management system, including:

[0112] A GPU resource pool construction module is used to divide the physical GPU graphics card resources into multiple GPU computing units according to the GPU division rules; and to build a GPU resource pool based on the GPU computing units;

[0113] A GPU task queue construction module is used to monitor the operating status information of the GPU resource pool in real time, obtain task parameters in the GPU resource pool according to the operating status information, and construct a GPU task queue with priority according to the task parameters;

[0114] The GPU task scheduling module dynamically allocates the tasks to be scheduled in the GPU task queue and the GPU computing power unit based on the resource scheduling strategy, and performs resource scheduling management on the GPU task queue based on the resource parameter information of the GPU computing power unit.

[0115] The present invention provides a GPU task queue management method, which manages the GPU queue through resource scheduling strategy and task priority. It can dynamically allocate GPU resources according to the actual needs of the tasks, ensure that key tasks can obtain the required resources in a timely manner, improve the overall scheduling efficiency, avoid resource waste and idleness, improve the management and scheduling efficiency of GPU resources, and realize the maximum value utilization of the GPU resource pool.

[0116] The present application also provides a GPU task queue management device, the device comprising:

[0117] a memory storing executable program code;

[0118] a processor coupled to the memory;

[0119] The processor calls the executable program code stored in the memory to execute the GPU task queue management method described above.

[0120] An embodiment of the present invention further provides a non-transitory machine-readable storage medium having an executable program stored thereon. When the executable program is executed by a processor, the processor is caused to execute the method provided in the above embodiment.

[0121] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the described method.

[0122] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute the described method.

[0123] The embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, and may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected based on actual needs to achieve the objectives of the embodiments. Those skilled in the art will be able to understand and implement the embodiments without inventive effort.

[0124] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by means of hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0125] Finally, it should be noted that the methods and systems disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A GPU task queue management method, characterized in that: The method comprises: Divide the physical GPU graphics card resources into multiple GPU computing power units according to the GPU partitioning rule; the GPU computing power units are distinguished according to resource parameter information, wherein the resource parameter information includes the GPU graphics card manufacturer name, GPU graphics card type, unit type, GPU computing power amount and / or GPU memory amount; Building a GPU resource pool based on the GPU computing power unit; Monitor the running status information of the GPU resource pool in real time, obtain task parameters in the GPU resource pool according to the running status information, and build a task queue with priority according to the task parameters; Dynamically allocate the tasks to be scheduled in the GPU task queue and the GPU computing power unit based on the resource scheduling strategy, and perform resource scheduling management on the GPU task queue based on the resource parameter information of the GPU computing power unit; The resource scheduling strategy includes: Determine the optimal execution task and the optimal GPU computing unit based on the task parameters of the task to be scheduled and the resources of the GPU computing unit, and perform task allocation; the task parameters include task priority and the shortest task execution time and waiting time; When a new task is received, the new task is inserted into any waiting task queue to be scheduled and queued for scheduling. The GPU computing unit in the current GPU resource pool is regularly checked to see whether there are idle resources available. The optimal execution task and the optimal GPU computing unit are determined based on the task parameters of the waiting task and the resource parameter information of the GPU computing unit. Determining the optimal execution task and the optimal GPU computing unit based on the task parameters of the task to be scheduled and the resources of the GPU computing unit also includes: Determine the task priority, minimum execution time, waiting time, number of required computing units, and elastic scaling type of the task to be scheduled, and determine the matching idle GPU computing units; When there are multiple tasks to be scheduled that match the same idle GPU computing unit, the optimal execution task of the optimal GPU computing unit is determined based on the matching distance and priority; Performing resource scheduling management on the GPU task queue based on the resource parameter information includes: Traverse the currently executing task queue, update the task status that meets the end conditions, and release the corresponding GPU resources to the GPU resource pool; Traverse the GPU resource pool, match the idle GPU resources according to the task parameters of the task to be scheduled, allocate the corresponding GPU computing power unit, and start the corresponding task process; If there are still tasks to be scheduled in the task queue, determine the tasks that meet the end conditions, update the status of the tasks that meet the end conditions, and release the corresponding GPU resources to the GPU resource pool; Allocate the corresponding resources to the task to be scheduled, and put the task that meets the end conditions into the queue of tasks to be scheduled.

2. A GPU task queue management method according to claim 1, characterized in that: GPU computing power units include hard partitioning based on MIG and VTPU ​​technologies, soft partitioning based on CUDA, RoCm, and oneAPI programming frameworks, and resource-sharing computing power units.

3. A GPU task queue management method according to claim 2, characterized in that: The termination condition is that the task has been completed or the actual running time of the task exceeds the preset running time.

4. A GPU task queue management system, characterized in that: include: A GPU resource pool construction module is configured to divide physical GPU graphics card resources into multiple GPU computing units according to GPU partitioning rules; the GPU computing units are distinguished according to resource parameter information, wherein the resource parameter information includes the GPU graphics card manufacturer name, GPU graphics card type, unit type, GPU computing power quantity and / or GPU memory quantity; and a GPU resource pool is constructed based on the GPU computing units; A GPU task queue construction module is used to monitor the operating status information of the GPU resource pool in real time, obtain task parameters in the GPU resource pool according to the operating status information, and construct a GPU task queue with priority according to the task parameters; The GPU task scheduling module dynamically allocates the tasks to be scheduled in the GPU task queue and the GPU computing power unit based on the resource scheduling strategy, and performs resource scheduling management on the GPU task queue based on the resource parameter information of the GPU computing power unit; The resource scheduling strategy includes: Determine the optimal execution task and the optimal GPU computing unit based on the task parameters of the task to be scheduled and the resources of the GPU computing unit, and perform task allocation; the task parameters include task priority and the shortest task execution time and waiting time; When a new task is received, the new task is inserted into any waiting task queue to be scheduled and queued for scheduling. The GPU computing unit in the current GPU resource pool is regularly checked to see whether there are idle resources available. The optimal execution task and the optimal GPU computing unit are determined based on the task parameters of the waiting task and the resource parameter information of the GPU computing unit. Determining the optimal execution task and the optimal GPU computing unit based on the task parameters of the task to be scheduled and the resources of the GPU computing unit also includes: Determine the task priority, minimum execution time, waiting time, number of required computing units, and elastic scaling type of the task to be scheduled, and determine the matching idle GPU computing units; When there are multiple tasks to be scheduled that match the same idle GPU computing unit, the optimal execution task of the optimal GPU computing unit is determined based on the matching distance and priority; Performing resource scheduling management on the GPU task queue based on the resource parameter information includes: Traverse the currently executing task queue, update the task status that meets the end conditions, and release the corresponding GPU resources to the GPU resource pool; Traverse the GPU resource pool, match the idle GPU resources according to the task parameters of the task to be scheduled, allocate the corresponding GPU computing power unit, and start the corresponding task process; If there are still tasks to be scheduled in the task queue, determine the tasks that meet the end conditions, update the status of the tasks that meet the end conditions, and release the corresponding GPU resources to the GPU resource pool; Allocate the corresponding resources to the task to be scheduled, and put the task that meets the end conditions into the queue of tasks to be scheduled.

5. A GPU task queue management device, characterized in that: The device comprises: a memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute a GPU task queue management method according to any one of claims 1 to 3.

Images