Cooperative scheduling method of heterogeneous GPU resources, electronic device and medium

Abstract

This invention relates to the field of resource scheduling technology, and more particularly to a collaborative scheduling method, electronic device, and medium for heterogeneous GPU resources. It achieves multi-mode resource delivery and a priority-driven scheduling mechanism by constructing a unified computing power pool across scenarios. This solution breaks down the physical and logical isolation between single-card development and multi-card computing scenarios, constructing a unified heterogeneous GPU resource pool to support cross-scenario resource sharing and collaborative scheduling. Furthermore, this invention provides multiple resource delivery forms such as containers, virtual machines, and bare metal, comprehensively covering diverse task requirements such as lightweight code debugging, custom environment deployment, and underlying driver development, significantly improving the platform's scenario adaptability and task coverage. This invention achieves intelligent collaborative scheduling for heterogeneous, multi-modal, and cross-pool GPU resources, improving the resource utilization and adaptability of heterogeneous GPU resource scheduling.

Description

Technical Field

[0001] This invention relates to the field of resource scheduling technology, and in particular to a collaborative scheduling method, electronic device, and medium for heterogeneous GPU resources. Background Technology

[0002] With the rapid development of artificial intelligence, high-performance computing (HPC), and hardware simulation technologies, graphics processing units (GPUs), as core computing power carriers, have been widely used in various scenarios such as model training, scientific computing, and driver development and debugging. However, current heterogeneous GPU resource scheduling technologies, which differ in architecture, model, manufacturer, and computing power specifications, have at least the following shortcomings: First, existing solutions generally adopt an independent resource deployment mode with scenario isolation, building separate GPU resource pools for single-card R&D scenarios and multi-card computing scenarios. The two are completely isolated physically and logically, making it impossible to achieve cross-scenario resource reuse and collaborative scheduling, resulting in low overall resource utilization and prominent computing resource silos.

[0003] Secondly, existing scheduling methods lack adaptability to heterogeneous GPU resources. As the hybrid deployment of domestic general-purpose computing GPs and mainstream international GPUs becomes a trend, traditional scheduling systems lack the ability to uniformly abstract and schedule heterogeneous GPUs. They cannot dynamically match suitable heterogeneous GPU resources according to the computing power requirements, environmental dependencies, and performance requirements of tasks, thus limiting the scalability of computing clusters.

[0004] Furthermore, existing scheduling methods offer limited resource delivery options, failing to meet diverse task requirements. Current platforms typically support only single delivery methods such as containers or bare metal, making it difficult to simultaneously satisfy the needs of lightweight code debugging for rapid startup and high-density deployment, the need for customized operating systems and drivers in virtual machine environments, and the need for complete hardware control for underlying driver development.

[0005] In addition, the existing scheduling mechanism lacks priority-driven ordered task queue management, and cannot effectively distinguish the resource preemption logic of high-priority training tasks, medium-priority debugging tasks and low-priority inference tasks. When resources are scarce, critical tasks are easily blocked and non-critical tasks occupy core resources, which cannot guarantee the stable operation of core business.

[0006] Therefore, there is an urgent need for an intelligent collaborative scheduling method for heterogeneous GPU resources that supports the delivery of resources in multiple forms, such as containers, virtual machines, and bare metal, to achieve cross-scenario resource reuse, heterogeneous GPU compatibility, and fine-grained computing power scheduling. This would solve the technical problems of low resource utilization, severe scene isolation, and insufficient adaptability in the scheduling of heterogeneous GPU resources in the existing technology, and achieve intelligent collaborative scheduling for heterogeneous, multi-modal, and cross-pool GPU resources. Summary of the Invention

[0007] The purpose of this invention is to provide a method, electronic device, and medium for collaborative scheduling of heterogeneous GPU resources, which realizes intelligent collaborative scheduling for heterogeneous, multimodal, and cross-pool GPU resources, and improves the resource utilization and adaptability of heterogeneous GPU resource scheduling.

[0008] According to a first aspect of the present invention, a method for collaborative scheduling of heterogeneous GPU resources is provided, comprising: Step S1: Set up a virtual machine resource pool, a single-card container resource pool, and a bare metal workstation resource pool for single-card R&D tasks; and set up a multi-card container resource pool and a bare metal cluster resource pool for multi-card computing tasks. Step S2: Obtain the task to be processed. For a single-card R&D task, execute step S3; for a multi-card computing task, execute step S9. Step S3: If the required resource delivery form is a virtual machine, then proceed to step S4; if the required resource delivery form is bare metal, then proceed to step S5; if the required resource delivery form is a container, then proceed to step S7. Step S4: If the virtual machine resource pool contains the GPU resources required for the task to be processed, then create a virtual machine instance based on the required GPU resources and return to step S2; otherwise, return directly to step S2. Step S5: If the bare metal workstation resource pool contains the GPU resources required by the task to be processed, then allocate a bare metal workstation to the task to be processed based on the required GPU resources and return to step S2; otherwise, proceed to step S6. Step S6: Allocate an idle bare metal node from the bare metal cluster resource pool as a bare metal workstation to the task to be processed, and return to step S2; Step S7: If the single-card container resource pool contains the GPU resources required by the task to be processed, then allocate a container node to the task to be processed based on the required GPU resources and return to step S2; otherwise, proceed to step S8. Step S8: Allocate an idle container node from the multi-card container resource pool to the task to be processed, and return to step S2; Step S9: If there are GPU resources allocated to a single-card R&D task that are required for the task to be processed, then first reclaim the corresponding allocated GPU resources and then execute step S10; otherwise, execute step S10 directly. Step S10: Allocate corresponding GPU resources for the task to be processed from the multi-GPU resource pool according to the resource delivery form of the task to be processed, and then return to step S2. If the GPU resources are insufficient, return directly to step S2.

[0009] According to a second aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in the first aspect of the present invention.

[0010] According to a third aspect of the present invention, a computer-readable storage medium is provided, storing computer-executable instructions for performing the method described in the first aspect of the present invention.

[0011] Compared with existing technologies, this invention has significant advantages and beneficial effects. Through the above technical solution, the collaborative scheduling method, electronic device, and medium for heterogeneous GPU resources provided by this invention achieve considerable technological advancement and practicality, and have broad industrial application value. It has at least the following beneficial effects: This invention effectively solves the problems of scenario isolation, low resource utilization, insufficient heterogeneous adaptability, and disordered task scheduling in existing technologies by constructing a unified computing power pool across scenarios and realizing multi-form resource delivery and priority-driven scheduling mechanisms. This solution breaks down the physical and logical isolation between single-card development and multi-card computing scenarios, constructing a unified heterogeneous GPU resource pool to support cross-scenario resource sharing and collaborative scheduling, eliminating computing power resource silos, and significantly improving overall resource utilization. Furthermore, this invention provides multiple resource delivery forms such as containers, virtual machines, and bare metal, comprehensively covering diverse task requirements such as lightweight code debugging, custom environment deployment, and underlying driver development, significantly improving the platform's scenario adaptability and task coverage. This invention achieves intelligent collaborative scheduling for heterogeneous, multi-modal, and cross-pool GPU resources, improving the resource utilization and adaptability of heterogeneous GPU resource scheduling. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating the collaborative scheduling method for heterogeneous GPU resources provided in an embodiment of the present invention; Figure 2This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a computer-readable storage medium structure provided for an embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] This invention provides a method for the collaborative scheduling of heterogeneous GPU resources, such as... Figure 1 As shown, it includes: Step S1: Set up a virtual machine resource pool, a single-card container resource pool, and a bare metal workstation resource pool for single-card R&D tasks, and set up a multi-card container resource pool and a bare metal cluster resource pool for multi-card computing tasks.

[0016] Single-GPU R&D tasks refer to tasks within a single-GPU R&D scenario. Multi-GPU computing tasks refer to tasks within a multi-level, multi-GPU training and inference scenario. A bare-metal workstation refers to a high-performance single-machine device with an operating system deployed directly on physical hardware, without virtualization or container layer encapsulation. A bare-metal cluster refers to a cluster of devices that interconnect multiple bare-metal workstations via a high-speed network, forming a unified resource pool for unified management, scheduling, and on-demand allocation. The virtual machine resource pool, single-GPU container resource pool, and bare-metal workstation resource pool constitute the R&D GPU resource pool; for multi-GPU computing tasks, the multi-GPU container resource pool and bare-metal cluster resource pool constitute the multi-GPU computing resource pool. By default, the R&D GPU resource pool and the multi-GPU computing GPU resource pool are physically isolated to avoid mutual interference. However, when the single-GPU container resource pool and bare-metal workstation resource pool for a single-GPU R&D task are insufficient to meet application requirements, they can be dynamically borrowed to achieve elastic scheduling.

[0017] Step S2: Obtain the task to be processed. For a single-card R&D task, execute step S3. For a multi-card computing task, execute step S9.

[0018] This involves creating a task profile for each pending task. The task profile can include information such as GPU brand, GPU model, number of GPUs, expected duration, whether Remote Direct Memory Access (RDMA) is required, computational precision requirements, preemptibility, and priority. Based on the task profile, the task type, resource delivery method, and required GPU resources are determined. During the acquisition of GPU resources for a task, the performance of heterogeneous GPUs can be abstracted into a preset standard computing unit and optimally matched according to task requirements. Guiding tasks to the most suitable resource pool based on their characteristics improves overall resource scheduling efficiency.

[0019] Step S3: If the required resource delivery form is a virtual machine, then proceed to step S4; if the required resource delivery form is bare metal, then proceed to step S5; if the required resource delivery form is a container, then proceed to step S7.

[0020] Step S4: If the virtual machine resource pool contains the GPU resources required for the task to be processed, then create a virtual machine instance based on the required GPU resources and return to step S2; otherwise, return directly to step S2.

[0021] Step S5: If the bare metal workstation resource pool contains the GPU resources required by the task to be processed, then allocate a bare metal workstation to the task to be processed based on the required GPU resources and return to step S2; otherwise, proceed to step S6.

[0022] Step S6: Allocate an idle bare metal node from the bare metal cluster resource pool as a bare metal workstation to the task to be processed, and return to step S2.

[0023] Step S7: If the single-card container resource pool contains the GPU resources required by the task to be processed, then allocate a container node to the task to be processed based on the required GPU resources and return to step S2; otherwise, proceed to step S8.

[0024] Step S8: Allocate an idle container node from the multi-card container resource pool to the task to be processed, and return to step S2.

[0025] Step S9: If there are GPU resources allocated to a single-card R&D task that are required for the task to be processed, then first reclaim the corresponding allocated GPU resources and then execute step S10; otherwise, execute step S10 directly.

[0026] Step S10: Allocate corresponding GPU resources for the task to be processed from the multi-GPU resource pool according to the resource delivery form of the task to be processed, and then return to step S2. If the GPU resources are insufficient, return directly to step S2.

[0027] As one embodiment, step S2, obtaining the task to be processed, includes: Step S21: Construct virtual machine resource pool task queues, single-card container resource pool task queues, bare metal workstation resource pool task queues, multi-card container resource pool task queues, and bare metal cluster resource pool task queues.

[0028] Step S22: Add the single-card R&D task with the required resource delivery form of virtual machine to the virtual machine resource pool task queue; add the single-card R&D task with the required resource delivery form of container to the single-card container resource pool task queue; add the single-card R&D task with the required resource delivery form of bare metal to the bare metal workstation resource pool task queue; add the multi-card computing task with the required resource delivery form of container to the multi-card container resource pool task queue; and add the multi-card computing task with the required resource delivery form of bare metal to the bare metal cluster resource pool task queue.

[0029] It should be noted that tasks can be directly inserted into the corresponding queue in chronological order, or the priority weight of the task can be determined based on the task profile, and the corresponding queue can be dynamically adjusted according to the priority weight.

[0030] Step S23: Obtain one task from each of the virtual machine resource pool task queue, single-card container resource pool task queue, bare metal workstation resource pool task queue, multi-card container resource pool task queue, and bare metal cluster resource pool task queue as a task to be processed.

[0031] It should be noted that the virtual machine resource pool, single-card container resource pool, bare metal workstation resource pool, multi-card container resource pool, and bare metal cluster resource pool are set up independently. Therefore, a task can be obtained from the task queue of the virtual machine resource pool, single-card container resource pool, bare metal workstation resource pool, multi-card container resource pool, and bare metal cluster resource pool as a task to be processed, and resources are allocated through the corresponding resource pool.

[0032] As one embodiment, in step S22, the bare metal workstation resource pool task queue and the bare metal cluster resource pool task queue are generated based on a preset bare metal scheduling method. The preset bare metal scheduling method includes: Step S100: Obtain the task to be added, wherein the task to be added is a bare metal workstation resource pool task or a bare metal cluster resource pool task.

[0033] Step S200: If the task to be added is in instant acquisition mode and the duration is less than the first time threshold, or the task to be added is in first time reservation mode, the actual start time from the current time is greater than the second time threshold and the reservation duration is less than the third time threshold, or the task to be added is in second time reservation mode, the time from the start time is greater than the fourth time threshold and the reservation duration is less than the fifth time threshold, then proceed to step S300. Otherwise, generate a prompt message and skip the task to be added, where the fourth time threshold > the second time threshold > the first time threshold, and the fifth time threshold > the third time threshold.

[0034] As an example, the instant access mode has a duration of less than 24 hours. The first-time reservation mode requires a start date of one day later and a reservation duration of less than 5 days. The second-time reservation mode requires 7 days in advance and a reservation duration of less than 6 months. By setting scheduling modes such as instant access, first-time reservation, and second-time reservation, the instant leasing and long-term reservation needs of bare metal in different business scenarios can be met.

[0035] Step S300: Based on the creation time, scheduling mode, business information and basic weight of the task to be added, obtain the priority weight of the task to be added.

[0036] The business information includes GPU type, delivery method, start time, end time, status information, node requirements, and corresponding node IP address information.

[0037] Step S400: Add the task to be added to the corresponding target queue. If the task to be added is a bare metal workstation resource pool task, the target queue is the bare metal workstation resource pool task queue. If the task to be added is a bare metal cluster resource pool task, the target queue is the bare metal cluster resource pool task queue.

[0038] Specifically, the target queue can be a Redis queue.

[0039] As one embodiment, step S300 includes: Step S301: Create an information record for the task to be added. The information record includes GPU type, delivery method, scheduling mode, start time, end time, priority weight, creation time, status information, node requirements and corresponding node IP address information. The priority weight is initially set to the base weight.

[0040] The delivery methods include single-machine 8-card and multi-machine 16-card configurations. Scheduling modes include instant acquisition, first-time reservation, and second-time reservation. Status information includes queuing status, running status, and completed status. Node requirements include single-node and multi-node requirements. It should be noted that if the delivery method is single-machine, the corresponding node requirement is a single-ground requirement; if the delivery method is multi-machine, the delivery method is multi-node. Ultimately, resource delivery is based on the corresponding node IP address.

[0041] Step S302: Generate the priority weight of the task to be added based on GPU type, delivery form, scheduling mode, start time, end time, basic weight, creation time, status information, node requirements and corresponding node IP address information.

[0042] It should be noted that, depending on different application requirements, different weights can be set for different information of the task to be added, and the priority weight of the task to be added can be obtained based on the different weights of different information.

[0043] As one embodiment, step S22 further includes: Step S500: When a task in the target queue is successfully allocated the corresponding bare metal resource, the task is deleted from the target queue.

[0044] Step S600: If a priority adjustment instruction is received, the target queue is adjusted based on the priority adjustment instruction.

[0045] Step S700: If some nodes of a multi-node task have already been assigned, increase the priority weight of the task by a preset weight increment, and then update the target queue.

[0046] It should be noted that step S700 can significantly increase the priority of this task when the node requirement involves multiple node allocations, ensuring that the remaining nodes are acquired as quickly as possible. The target queue can be dynamically adjusted through steps S500-S700.

[0047] As one embodiment, step S6 includes: Step S61: Determine whether there are any idle bare metal resources in the bare metal cluster resource pool. If there are, proceed to step S62; otherwise, return directly to step S2.

[0048] Step S62: Allocate an idle bare metal node from the bare metal cluster resource pool, set a first allocation tag for the allocated bare metal node, assign the allocated bare metal node as a bare metal workstation to the task to be processed, return to step S2, after the task to be processed is completed, remove the first allocation tag of the allocated bare metal node, and return the allocated bare metal node to the multi-card container resource pool.

[0049] Specifically, step S62 can be implemented as follows: First, a bare metal node is dynamically prepared from the bare metal cluster resource pool using the Bare Metal Operator (BMO) tool. Second, the node is added to a preset "R&D Temporary Rental" node group for management. Third, user tasks or containers are scheduled to run on the bare metal node using Kubernetes (K8s) label selector or node affinity mechanism. Finally, after the task is completed, the node is automatically deprovisioned and returned to the computing cluster resource pool, achieving cyclic reuse and efficient scheduling of node resources.

[0050] As one embodiment, step S8 includes: Step S81: Determine whether there are any unoccupied and unlocked container nodes in the multi-card container resource pool. If so, proceed to step S82; otherwise, return directly to step S2.

[0051] Step S82: Allocate an idle container node from the multi-card container resource pool, set a second allocation tag and taint for the allocated container node, configure the corresponding taint tolerance for the task to be processed, then assign the allocated container node to the task to be processed, return to step S2, after the task to be processed is completed, remove the second allocation tag and taint from the allocated container node, and return the allocated container node to the multi-card container resource pool.

[0052] Specifically, step S82 can be implemented as follows: use the Kubernetes extended scheduler or a custom scheduler plugin to identify idle nodes, and control the admission and resource occupation of borrowed tasks through node taints and tolerance, resource quotas and priority mechanisms; and in conjunction with the descheduler, automatically evict borrowed tasks when the original computing task is restored, thereby achieving elastic reuse of GPU resources without affecting core business.

[0053] As one embodiment, step S10 includes: Step S101: If the resource delivery form of the task to be processed is a container, then execute step S102; if the resource delivery form of the task to be processed is bare metal, then execute step S103. Step S102: If the multi-GPU container resource pool contains the GPU resources required by the task to be processed, then allocate a multi-GPU container node to the task to be processed based on the required GPU resources and return to step S2; otherwise, return directly to step S2. Step S103: If the bare metal cluster resource pool has the GPU resources required by the task to be processed, then allocate the bare metal cluster to the task to be processed based on the required GPU resources and return to step S2; otherwise, return directly to step S2.

[0054] It should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. A process can be terminated when its operation is complete, but it may also have additional steps not included in the figures. A process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0055] This invention effectively solves the problems of scenario isolation, low resource utilization, insufficient heterogeneous adaptability, and disordered task scheduling in existing technologies by constructing a unified computing power pool across scenarios and implementing a multi-form resource delivery and priority-driven scheduling mechanism. This solution breaks down the physical and logical isolation between single-card development and multi-card computing scenarios, constructing a unified heterogeneous GPU resource pool to support cross-scenario resource sharing and collaborative scheduling, eliminating computing power resource silos, and significantly improving overall resource utilization. Furthermore, this invention provides multiple resource delivery forms such as containers, virtual machines, and bare metal, comprehensively covering diverse task requirements such as lightweight code debugging, custom environment deployment, and underlying driver development, significantly improving the platform's scenario adaptability and task coverage. This invention achieves intelligent collaborative scheduling for heterogeneous, multi-modal, and cross-pool GPU resources, improving the resource utilization and adaptability of heterogeneous GPU resource scheduling.

[0056] This invention also provides an electronic device, such as... Figure 2 As shown, it includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in the embodiments of the present invention.

[0057] This invention also provides a computer-readable storage medium, such as... Figure 3 As shown, computer-executable instructions are stored thereon, which are used to execute the methods described in the embodiments of the present invention.

[0058] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for collaborative scheduling of heterogeneous GPU resources, characterized in that, include: Step S1: Set up a virtual machine resource pool, a single-card container resource pool, and a bare metal workstation resource pool for single-card R&D tasks; and set up a multi-card container resource pool and a bare metal cluster resource pool for multi-card computing tasks. Step S2: Obtain the task to be processed. For a single-card R&D task, execute step S3; for a multi-card computing task, execute step S9. Step S3: If the required resource delivery form is a virtual machine, then proceed to step S4; if the required resource delivery form is bare metal, then proceed to step S5; if the required resource delivery form is a container, then proceed to step S7. Step S4: If the virtual machine resource pool contains the GPU resources required for the task to be processed, then create a virtual machine instance based on the required GPU resources and return to step S2; otherwise, return directly to step S2. Step S5: If the bare metal workstation resource pool contains the GPU resources required by the task to be processed, then allocate a bare metal workstation to the task to be processed based on the required GPU resources and return to step S2; otherwise, proceed to step S6. Step S6: Allocate an idle bare metal node from the bare metal cluster resource pool as a bare metal workstation to the task to be processed, and return to step S2; Step S7: If the single-card container resource pool contains the GPU resources required by the task to be processed, then allocate a container node to the task to be processed based on the required GPU resources and return to step S2; otherwise, proceed to step S8. Step S8: Allocate an idle container node from the multi-card container resource pool to the task to be processed, and return to step S2; Step S9: If there are GPU resources allocated to a single-card R&D task that are required for the task to be processed, then first reclaim the corresponding allocated GPU resources and then execute step S10; otherwise, execute step S10 directly. Step S10: Allocate corresponding GPU resources for the task to be processed from the multi-GPU resource pool according to the resource delivery form of the task to be processed, and then return to step S2. If the GPU resources are insufficient, return directly to step S2.

2. The method according to claim 1, characterized in that: In step S2, obtaining the task to be processed includes: Step S21: Construct virtual machine resource pool task queues, single-card container resource pool task queues, bare metal workstation resource pool task queues, multi-card container resource pool task queues, and bare metal cluster resource pool task queues; Step S22: Add the single-card R&D task with the required resource delivery form of virtual machine to the virtual machine resource pool task queue; add the single-card R&D task with the required resource delivery form of container to the single-card container resource pool task queue; add the single-card R&D task with the required resource delivery form of bare metal to the bare metal workstation resource pool task queue; add the multi-card computing task with the required resource delivery form of container to the multi-card container resource pool task queue; and add the multi-card computing task with the required resource delivery form of bare metal to the bare metal cluster resource pool task queue. Step S23: Obtain one task from each of the virtual machine resource pool task queue, single-card container resource pool task queue, bare metal workstation resource pool task queue, multi-card container resource pool task queue, and bare metal cluster resource pool task queue as a task to be processed.

3. The method according to claim 2, characterized in that: In step S22, the bare metal workstation resource pool task queue and the bare metal cluster resource pool task queue are generated based on a preset bare metal scheduling method. The preset bare metal scheduling method includes: Step S100: Obtain the task to be added, wherein the task to be added is a bare metal workstation resource pool task or a bare metal cluster resource pool task. Step S200: If the task to be added is in instant acquisition mode and the duration is less than the first time threshold, or the task to be added is in first time reservation mode, the actual start time from the current time is greater than the second time threshold and the reservation duration is less than the third time threshold, or the task to be added is in second time reservation mode, the time from the start time is greater than the fourth time threshold and the reservation duration is less than the fifth time threshold, then proceed to step S300; otherwise, generate a prompt message and skip the task to be added, where the fourth time threshold > the second time threshold > the first time threshold, and the fifth time threshold > the third time threshold. Step S300: Based on the creation time, scheduling mode, business information and basic weight of the task to be added, obtain the priority weight of the task to be added; Step S400: Add the task to be added to the corresponding target queue. If the task to be added is a bare metal workstation resource pool task, the target queue is the bare metal workstation resource pool task queue. If the task to be added is a bare metal cluster resource pool task, the target queue is the bare metal cluster resource pool task queue.

4. The method according to claim 3, characterized in that: Step S300 includes: Step S301: Create an information record for the task to be added. The information record includes GPU type, delivery method, scheduling mode, start time, end time, priority weight, creation time, status information, node requirements and corresponding node IP address information. The priority weight is initially set to the basic weight. Step S302: Generate the priority weight of the task to be added based on GPU type, delivery form, scheduling mode, start time, end time, basic weight, creation time, status information, node requirements and corresponding node IP address information.

5. The method according to claim 3, characterized in that: Step S22 also includes: Step S500: When a task in the target queue is successfully allocated the corresponding bare metal resource, the task is deleted from the target queue. Step S600: If a priority adjustment instruction is received, adjust the target queue based on the priority adjustment instruction; Step S700: If some nodes of a multi-node task have already been assigned, increase the priority weight of the task by a preset weight increment, and then update the target queue.

6. The method according to claim 1, characterized in that: Step S6 includes: Step S61: Determine whether there are any idle bare metal resources in the bare metal cluster resource pool. If there are, proceed to step S62; otherwise, return directly to step S2. Step S62: Allocate an idle bare metal node from the bare metal cluster resource pool, set a first allocation tag for the allocated bare metal node, assign the allocated bare metal node as a bare metal workstation to the task to be processed, return to step S2, after the task to be processed is completed, remove the first allocation tag of the allocated bare metal node, and return the allocated bare metal node to the multi-card container resource pool.

7. The method according to claim 1, characterized in that: Step S8 includes: Step S81: Determine whether there are any unoccupied and unlocked container nodes in the multi-card container resource pool. If there are, proceed to step S82; otherwise, return directly to step S2. Step S82: Allocate an idle container node from the multi-card container resource pool, set a second allocation tag and taint for the allocated container node, configure the corresponding taint tolerance for the task to be processed, then assign the allocated container node to the task to be processed, return to step S2, after the task to be processed is completed, remove the second allocation tag and taint from the allocated container node, and return the allocated container node to the multi-card container resource pool.

8. The method according to claim 1, characterized in that: Step S10 includes: Step S101: If the resource delivery form of the task to be processed is a container, then execute step S102; if the resource delivery form of the task to be processed is bare metal, then execute step S103. Step S102: If the multi-GPU container resource pool contains the GPU resources required by the task to be processed, then allocate a multi-GPU container node to the task to be processed based on the required GPU resources and return to step S2; otherwise, return directly to step S2. Step S103: If the bare metal cluster resource pool has the GPU resources required by the task to be processed, then allocate the bare metal cluster to the task to be processed based on the required GPU resources and return to step S2; otherwise, return directly to step S2.

9. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that are executed by the at least one processor, the instructions being configured to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The device stores computer-executable instructions for performing the method as described in any one of claims 1-8.

Images