A GPU resource scheduling method and a computer readable storage medium

Abstract

This invention discloses a GPU resource scheduling method and a computer-readable storage medium. The method includes: obtaining a GPU resource declaration carried in a target Pod; wherein the GPU resource declaration includes multiple GPU resource configuration items arranged in descending order of priority; sequentially parsing each GPU resource configuration item to obtain a specification object corresponding to each GPU resource configuration item; traversing the parsed specification objects in descending order of priority, and attempting to filter out target nodes that satisfy the currently traversed specification objects in the target cluster where GPUs are deployed; if the target node is selected, binding the target Pod to the target node and allocating GPU resources to the target Pod according to the currently traversed specification object; if the target node is not selected, continuing to traverse the parsed specification objects until the target node is selected or the traversal is completed.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a GPU resource scheduling method and a computer-readable storage medium. Background Technology

[0002] With the widespread application of deep learning and artificial intelligence technologies, graphics processing units (GPUs) have become the core computing power resource for machine learning training and inference tasks. In enterprise-level AI computing platforms, the demand for GPU resources for different types of AI tasks exhibits significant heterogeneity: lightweight inference tasks typically require only a small amount of video memory, while large-scale model training tasks require multiple high-performance GPU cards to obtain sufficient computing power and video memory support.

[0003] Currently, mainstream GPU resource scheduling technologies in the industry typically only support a single mode, either a purely exclusive mode or a purely shared mode, leading to the following problems:

[0004] Defect 1: Limited scheduling success rate and lack of flexibility.

[0005] When high-priority primary resources (such as H100 full cards) are temporarily scarce, the existing scheduler cannot automatically detect and downgrade to match secondary resources (such as A800 full cards or 3090 shared memory), and can only place tasks in the waiting queue indefinitely. Actual test data shows that during periods of resource scarcity, the scheduling success rate in pure exclusive mode is only about 65%, severely impacting business timeliness.

[0006] Defect 2: Resource utilization is polarized.

[0007] In dedicated clusters, small tasks consume large GPUs, resulting in extremely low utilization; in shared clusters, large model training tasks suffer performance degradation because they cannot obtain physical exclusivity. Enterprises are forced to build multiple physically isolated clusters, significantly increasing hardware procurement costs and operational complexity.

[0008] Defect 3: Fragmented configuration semantics, resulting in a poor user experience.

[0009] The lack of a unified declarative syntax means that users need to manually modify task configurations based on cluster type.

[0010] There is currently no effective solution to the aforementioned shortcomings of existing technologies. Summary of the Invention

[0011] The purpose of this invention is to provide a GPU resource scheduling method and a computer-readable storage medium. Through the design of a unified declarative syntax, it supports defining a hybrid strategy of exclusive or shared GPU cards in a single configuration, and can automatically match based on priority degradation. This invention was completed with the support of a project entitled "Key Technologies for the Construction and Application of Large Models in Architectural Engineering," numbered "2024YFC3811200."

[0012] According to one aspect of the present invention, a GPU resource scheduling method is provided, comprising: Obtain the GPU resource declaration carried in the target Pod; wherein, the GPU resource declaration includes multiple GPU resource configuration items arranged in descending order of priority; Each GPU resource configuration item is parsed sequentially to obtain the specification object corresponding to each GPU resource configuration item; among them, the exclusive specification object used to represent the exclusive GPU includes the GPU type and the number of GPUs of that type, and the shared specification object used to represent the shared GPU includes the GPU type and the video memory capacity. The parsed specification objects are traversed in descending order of priority, and target nodes that meet the specification objects currently being traversed are selected from the target clusters where GPUs are deployed. If the target node is selected, the target Pod is bound to the target node, and GPU resources are allocated to the target Pod according to the specification object currently traversed. If the target node is not found, continue traversing the parsed specification objects until the target node is found or the traversal is complete.

[0013] Optionally, the step of sequentially parsing each GPU resource configuration item to obtain the specification object corresponding to each GPU resource configuration item includes: Parse the mode identifier, GPU model, and resource quantity values ​​carried in the GPU resource configuration item; If the pattern identifier is an exclusive identifier used to characterize an exclusive GPU, then the data format associated with the exclusive identifier is obtained, the GPU model is used as the GPU type in the data format, and the resource quantity value is used as the number of GPUs in the data format to construct the exclusive specification object. If the pattern identifier is a shared identifier used to characterize a shared GPU, then the data format associated with the shared identifier is obtained, the GPU model is used as the GPU type in the data format, and the resource quantity value is used as the video memory capacity in the data format to construct the shared specification object.

[0014] Optionally, the step of traversing the parsed specification objects in descending order of priority and attempting to filter out target nodes that satisfy the currently traversed specification objects in the target cluster where GPUs are deployed includes: Obtain the GPU quota of the target tenant to which the target Pod belongs and the GPU quota of the target resource pool to which the target tenant belongs; wherein, the GPU quota includes the upper limit of the number of GPUs and the upper limit of the video memory capacity; If the current traversed specification object is an exclusive specification object, then check whether the sum of the number of GPUs used by the target tenant and the number of GPUs in the exclusive specification object exceeds the upper limit of the number of GPUs of the target tenant, and check whether the sum of the number of GPUs used by all tenants in the target resource pool and the number of GPUs in the exclusive specification object exceeds the upper limit of the number of GPUs of the target resource pool. If the currently traversed specification object is a shared specification object, then check whether the sum of the video memory capacity used by the target tenant and the video memory capacity in the shared specification object exceeds the upper limit of the video memory capacity of the target tenant, and check whether the sum of the video memory capacity used by all tenants in the target resource pool and the video memory capacity in the shared specification object exceeds the upper limit of the video memory capacity of the target resource pool. If all verifications pass, then try to filter out target nodes in the target cluster that meet the specifications of the currently traversed object; If any verification fails, it is determined that the target node has not been selected.

[0015] Optionally, if all verifications pass, then in the target cluster, an attempt is made to filter out target nodes that meet the specifications of the currently traversed object, including: Obtain the GPU type whitelist of the target resource pool; Determine whether the GPU type in the currently traversed specification object is within the GPU type whitelist; If the GPU type is not in the GPU type whitelist, then the target node has not been selected. If the GPU type is within the GPU type whitelist and the currently traversed specification object is an exclusive specification object, then try to filter out the target node that meets the exclusive specification object in the target cluster. If the GPU type is in the GPU type whitelist and the specification object currently being traversed is a shared specification object, then the upper limit of single memory allocation pre-configured for the GPU type by the target resource pool is obtained, and it is determined whether the memory capacity in the shared specification object is greater than the upper limit of single memory allocation. If the video memory capacity is greater than the upper limit of a single video memory request, then it is determined that the target node has not been selected. If the video memory capacity is less than or equal to the upper limit of the single video memory request, then an attempt will be made to filter out target nodes that meet the shared specification object in the target cluster.

[0016] Optionally, the step of traversing the parsed specification objects in descending order of priority and attempting to filter out target nodes that satisfy the currently traversed specification objects in the target cluster where GPUs are deployed includes: The system attempts to filter candidate nodes in the target cluster that satisfy the specifications of the currently traversed object. Each candidate node satisfies a preset hard constraint. If the currently traversed specification object is an exclusive specification object, the hard constraint is that the candidate node has at least one GPU card that matches the GPU type in the exclusive specification object and is in an idle state. If the currently traversed specification object is a shared specification object, the hard constraint is that the candidate node has at least one GPU card that matches the GPU type in the shared specification object, and the remaining video memory capacity of the GPU card is greater than or equal to the video memory capacity in the shared specification object. If no candidate nodes are found, then the target node has not been found. If a candidate node is selected, then that candidate node is taken as the target node; If multiple candidate nodes are selected, one candidate node is chosen from among them as the target node.

[0017] Optionally, if multiple candidate nodes are selected, selecting one candidate node from the multiple candidate nodes as the target node includes: If the specification object being traversed is an exclusive specification object, then determine the total number of GPUs belonging to the GPU type in the exclusive specification object in each candidate node, and determine the number of GPUs belonging to the GPU type and in a non-idle state in each candidate node. Calculate the ratio of the number of GPUs in a non-idle state to the total number of GPUs in each candidate node to obtain the load factor of each candidate node. The candidate node with the highest load factor is determined as the target node.

[0018] Optionally, if multiple candidate nodes are selected, selecting one candidate node from the multiple candidate nodes as the target node includes: If the current specification object being traversed is a shared specification object, then determine the total number of GPU types contained in all the parsed specification objects, and determine the priority position of the shared specification object among all the parsed specification objects. Based on the total number of GPU types and the priority ranking, the generational affinity score of the GPU type is determined; wherein, the generational affinity score is negatively correlated with the priority ranking and positively correlated with the total number of GPU types; Determine the total video memory capacity and remaining video memory capacity of all GPU cards belonging to this GPU type in each candidate node; Calculate the ratio of the remaining video memory capacity to the total video memory capacity in each candidate node to obtain the video memory fragmentation rate score for each candidate node. The intergenerational affinity score and memory fragmentation rate score of each candidate node are weighted and summed to obtain the comprehensive score of each candidate node. The candidate node with the highest overall score is determined as the target node.

[0019] Optionally, the target cluster includes multiple resource pools, each resource pool includes at least one tenant, each tenant includes multiple Pods, and each resource pool pre-sets a reserved requirement for each GPU type, the reserved requirement being used to indicate the number of GPUs that the resource pool wants to reserve for the corresponding GPU type; Each GPU type corresponds to a global reserved water level, which is the maximum value among all resource pools for the reserved demand of the corresponding GPU type; The GPU resources of the target cluster are pre-divided into a global buffer pool and a public resource pool. Both the global buffer pool and the public resource pool include multiple sub-pools. Each sub-pool corresponds to a GPU type. Each sub-pool includes GPU cards of the GPU type corresponding to that sub-pool, located on at least one node. The number of GPUs contained in each sub-pool of the global buffer pool is equal to the global reserved water level for the GPU type corresponding to each sub-pool. The number of GPUs contained in each sub-pool of the public resource pool is equal to the difference between the number of GPUs of the GPU type corresponding to each sub-pool in the target cluster and the global reserved water level.

[0020] Optionally, the step of traversing the parsed specification objects in descending order of priority and attempting to filter out target nodes that satisfy the currently traversed specification objects in the target cluster where GPUs are deployed includes: Determine the task type corresponding to the target Pod; If the task type is a reserved task used to represent core business, then the target node that meets the specification of the currently traversed object will be selected from the global buffer pool. If the task type is a common task used to represent non-core business, then the target node that meets the specifications of the currently traversed object will be selected from the public resource pool.

[0021] To achieve the above objectives, the present invention further provides a GPU resource scheduling device, comprising: The acquisition module is used to acquire the GPU resource declaration carried in the target Pod; wherein, the GPU resource declaration includes multiple GPU resource configuration items arranged in descending order of priority; The parsing module is used to parse each GPU resource configuration item in sequence to obtain the specification object corresponding to each GPU resource configuration item; among them, the exclusive specification object used to represent the exclusive GPU includes the GPU type and the number of GPUs of that type, and the shared specification object used to represent the shared GPU includes the GPU type and the video memory capacity. The filtering module is used to traverse the parsed specification objects in descending order of priority and attempt to filter out target nodes that meet the specification objects currently being traversed in the target cluster where GPUs are deployed. The scheduling module is used to bind the target Pod to the target node when the target node is selected, and to allocate GPU resources to the target Pod according to the specification object currently traversed. The determination module is used to continue traversing the parsed specification objects if the target node is not selected, until the target node is selected or the traversal is completed.

[0022] To achieve the above objectives, the present invention also provides a computer device, the computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the GPU resource scheduling method described above.

[0023] To achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the steps of the GPU resource scheduling method described above.

[0024] The GPU resource scheduling method and computer-readable storage medium provided by this invention establish a unified declarative syntax and priority traversal mechanism. Users only need to configure a declaration containing multiple GPU resource configuration items in the Pod to express their preference sequence for exclusive or shared resources. The scheduler automatically attempts to match resources according to priority order. When high-priority resources are insufficient, no manual intervention or task queuing is required. It automatically downgrades to try the next best resource, which significantly improves the scheduling success rate in resource-scarce scenarios and avoids tasks being blocked for a long time due to insufficient resources. At the same time, this mechanism breaks the limitation of the separation between exclusive and shared modes in traditional scheduling, allowing the same task to flexibly adapt to different allocation methods according to resource levels. This not only ensures the demand for exclusive resources for large model training, but also provides a shared fallback for lightweight tasks, thereby improving scheduling flexibility while optimizing the process. Attached Figure Description

[0025] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart of the GPU resource scheduling method provided in Example 1; Figure 2 This is a schematic diagram of the overall system architecture provided in Example 1; Figure 3 This is a core flowchart of GPU resource scheduling provided in Example 1; Figure 4 This is a schematic diagram of multi-tenant management of the resource pool provided in Example 1; Figure 5 This is a schematic diagram of the syntax parsing for the hybrid scheduling mode provided in Implementation Example 1; Figure 6 A block diagram of the GPU resource scheduling device provided in Embodiment 2; Figure 7 A block diagram of a computer device suitable for implementing the GPU resource scheduling method, provided in Embodiment 3. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0027] Example 1 Embodiment 1 of the present invention provides a GPU resource scheduling method, such as... Figure 1 As shown, the method includes steps S1 to S5, wherein: Step S1: Obtain the GPU resource declaration carried in the target Pod; wherein the GPU resource declaration includes multiple GPU resource configuration items arranged in descending order of priority.

[0028] Step S2: Parse each GPU resource configuration item in sequence to obtain the specification object corresponding to each GPU resource configuration item; wherein, the exclusive specification object used to represent the exclusive GPU includes the GPU type and the number of GPUs of that type, and the shared specification object used to represent the shared GPU includes the GPU type and the video memory capacity.

[0029] GPU resource declarations are included in Pod annotations. When a user needs to run a task that requires GPU resources in a Kubernetes cluster, they will include a GPU resource declaration in the submitted Pod annotation. This GPU resource declaration uses a unified syntax format to express the user's various preferences for GPU resources and their priority order. Each GPU resource configuration item is used to indicate either exclusive use of a certain number of GPU cards of a specific GPU type, or to indicate sharing a portion of the video memory capacity of a certain GPU type.

[0030] Optionally, each GPU resource configuration item includes a GPU model, a mode identifier, and a resource quantity value. When the mode identifier is an exclusive identifier used to represent a dedicated GPU, the resource quantity value is used to represent the number of GPU cards required; when the mode identifier is a shared identifier used to represent a shared GPU, the resource quantity value is used to represent the required video memory capacity.

[0031] For example, a GPU resource declaration can be represented as: "H100:2, A800 / 24, A10 / 16". Here, ":" represents the exclusive identifier, and " / " represents the shared identifier. H100:2 indicates the highest priority configuration, where the user expects to use two H100 GPUs in exclusive mode; A800 / 24 ​​indicates the next highest priority configuration, which, if the exclusive mode requirement cannot be met, will instead request 24GB of A800 video memory in shared mode; A10 / 16 indicates the lowest priority configuration, which, if neither of the first two can be met, will instead request 16GB of A10 video memory in shared mode.

[0032] Optionally, the step of sequentially parsing each GPU resource configuration item to obtain the specification object corresponding to each GPU resource configuration item includes: Parse the mode identifier, GPU model, and resource quantity values ​​carried in the GPU resource configuration item; If the pattern identifier is an exclusive identifier used to characterize an exclusive GPU, then the data format associated with the exclusive identifier is obtained, the GPU model is used as the GPU type in the data format, and the resource quantity value is used as the number of GPUs in the data format to construct the exclusive specification object. If the pattern identifier is a shared identifier used to characterize a shared GPU, then the data format associated with the shared identifier is obtained, the GPU model is used as the GPU type in the data format, and the resource quantity value is used as the video memory capacity in the data format to construct the shared specification object.

[0033] Specifically, during the parsing process, the order of GPU resource configuration items in the GPU resource declaration is preserved; that is, the specification objects parsed earlier correspond to higher priority, and those parsed later correspond to lower priority. The parsed specification objects can be stored in a priority queue in this order for subsequent scheduling. The unit of video memory capacity, such as GB, is preset in advance.

[0034] For example, taking the declaration "H100:2, A800 / 24, A10 / 16" in step S1 as an example, the parsing process is as follows: When parsing the first configuration item H100:2, the exclusive identifier ":" is identified, indicating an exclusive mode; the GPU model is H100, and the resource quantity is 2; an exclusive specification object is constructed: {GPU type: H100, GPU quantity: 2}.

[0035] When parsing the second configuration item A800 / 24, the shared identifier " / " is identified, indicating a shared mode; the GPU model is A800, and the resource value is 24; a shared specification object is constructed: {GPU type: A800, video memory capacity: 24}.

[0036] When parsing the third configuration item A10 / 16, the shared identifier " / " is identified, indicating a shared mode; the GPU model is A10, and the resource value is 16; a shared specification object is constructed: {GPU type: A10, video memory capacity: 16}.

[0037] After parsing, the scheduler obtains a priority queue containing three specification objects: [Exclusive {H100, 2}, Shared {A800, 24}, Shared {A10, 16}], and the queue order is the priority order specified by the user.

[0038] Through the parsing mechanism in this embodiment, the present invention achieves automated parsing of user-unified declarations, transforming abstract configuration strings into structured specification objects that can be directly used by the scheduler. This process not only accurately reproduces the user's various demands for GPU resources and their priority relationships, but also lays the foundation for subsequent automatic degradation scheduling, enabling the scheduler to attempt matching in sequence, thereby achieving elastic scheduling when resources are scarce.

[0039] Step S3: Traverse the parsed specification objects in descending order of priority, and attempt to filter out target nodes that meet the specifications of the currently traversed target nodes in the target cluster where GPUs are deployed.

[0040] Step S4: If the target node is selected, the target Pod is bound to the target node, and GPU resources are allocated to the target Pod according to the specification object currently traversed.

[0041] Step S5: If the target node is not found, continue traversing the parsed specification objects until the target node is found or the traversal is completed.

[0042] Specifically, according to the priority order specified by the user, try to match nodes that meet the resource requirements for the target Pod in turn. Once successful, the traversal is terminated and binding is performed, which is step S4. If it fails, it automatically downgrades to try the next priority specification object until it succeeds or all specification objects have been tried, which is step S5.

[0043] The scheduler maintains a queue of specification objects, ordered according to the priority specified by the user in the GPU resource declaration. The scheduler processes specification objects sequentially, starting from the head of the queue (i.e., the highest priority): First, retrieve the specification object of the current priority from the queue, denoted as currentSpec.

[0044] Secondly, attempt to filter target nodes: Based on the resource requirements defined in currentSpec (GPU type and number in exclusive mode, or GPU type and memory capacity in shared mode), perform a filtering operation on all nodes in the target cluster to find nodes that meet the resource requirements of currentSpec as target nodes. The filtering process must meet preset hard constraints, such as: If currentSpec is an exclusive specification object, then the candidate node must have at least a specified number of complete GPU cards that match the required GPU type and are in a completely idle state. If currentSpec is a shared specification object, then the candidate node must have at least one GPU card that matches the required GPU type, and the remaining video memory capacity of the card is not less than the required video memory capacity.

[0045] If one or more nodes that meet the conditions are successfully selected, one of them is selected as the target node (the specific selection method can be refined in subsequent steps), the traversal process is terminated, and the node binding and resource allocation stage is entered.

[0046] If no node that meets the criteria is found, the scheduling of the current specification object fails, and the scheduler moves the current pointer to the next lower priority specification object and repeats the above process.

[0047] When a specification object successfully identifies a target node, the traversal stops immediately, and subsequent lower-priority specification objects are no longer considered. The traversal ends when all specification objects have been tried without success, the scheduling attempt fails, and the Pod will remain in a waiting state until resources become available or the user cancels the request.

[0048] This embodiment introduces an automatic degradation mechanism. Traditional schedulers only support a single mode, and when the preferred resource is scarce, the Pod can only wait indefinitely. However, this invention, through the design of multi-priority specification objects, enables the scheduler to automatically and seamlessly degrade to try suboptimal resource solutions when high-priority resources are insufficient, thereby significantly improving the scheduling success rate.

[0049] For example, if a user-configured GPU resource declaration is "H100: 2, A800 / 24, A10 / 16", the scheduler first attempts to find a node with two free H100 GPUs. If H100 resources in the cluster are scarce and cannot meet the exclusive requirement, the scheduler will not block the Pod but will automatically downgrade to try the second priority: finding a node with at least one A800 GPU with at least 24GB of remaining video memory. If this is still not possible, it will continue to downgrade to try the third priority: finding a node with at least one A10 GPU with at least 16GB of remaining video memory. As long as any priority is successful, the Pod can run, avoiding long waits caused by insufficient primary resources.

[0050] Optionally, the step of traversing the parsed specification objects in descending order of priority and attempting to filter out target nodes that satisfy the currently traversed specification objects in the target cluster where GPUs are deployed includes: Obtain the GPU quota of the target tenant to which the target Pod belongs and the GPU quota of the target resource pool to which the target tenant belongs; wherein, the GPU quota includes the upper limit of the number of GPUs and the upper limit of the video memory capacity; If the current traversed specification object is an exclusive specification object, then check whether the sum of the number of GPUs used by the target tenant and the number of GPUs in the exclusive specification object exceeds the upper limit of the number of GPUs of the target tenant, and check whether the sum of the number of GPUs used by all tenants in the target resource pool and the number of GPUs in the exclusive specification object exceeds the upper limit of the number of GPUs of the target resource pool. If the currently traversed specification object is a shared specification object, then check whether the sum of the video memory capacity used by the target tenant and the video memory capacity in the shared specification object exceeds the upper limit of the video memory capacity of the target tenant, and check whether the sum of the video memory capacity used by all tenants in the target resource pool and the video memory capacity in the shared specification object exceeds the upper limit of the video memory capacity of the target resource pool. If all verifications pass, then try to filter out target nodes in the target cluster that meet the specifications of the currently traversed object; If any verification fails, it is determined that the target node has not been selected.

[0051] Specifically, before traversing the specification objects in priority order and attempting to filter target nodes, this embodiment first performs multi-tenant hierarchical quota verification to ensure, from a resource management perspective, that the resource request does not exceed the authorized scope of the tenant and its resource pool.

[0052] A tenant is a user or project group that submits a Pod, and is the basic unit of resource usage. A resource pool is a department or organizational unit to which a tenant belongs, containing one or more tenants, used to implement departmental-level total resource control. The GPU limit is the maximum number of GPU cards a tenant or resource pool can use, applicable to dedicated mode. The video memory limit is the maximum amount of video memory a tenant or resource pool can occupy, applicable to shared mode. The used video memory capacity is the amount of resources currently occupied by a tenant or resource pool, maintained in real time by the scheduler.

[0053] In existing technologies, the lack of an effective quota management mechanism in multi-tenant shared GPU clusters can lead to the following problems: excessive resource consumption by a single tenant, causing other tenants' tasks to fail; multiple tenants within the same resource pool competing for resources, resulting in resource allocation imbalance; and the lack of unified quota management for exclusive and shared resources, making it difficult to adapt to hybrid scheduling scenarios. To address this issue, this embodiment employs a two-layer verification mechanism at the tenant and resource pool levels to prevent individual tenants from overusing resources, ensuring fairness among tenants within the same resource pool; it also prevents entire departments from overusing resources, achieving resource isolation between departments and ensuring that the total resources of each department do not exceed the procurement or allocation limits; and it verifies the number of GPUs and memory capacity separately for exclusive and shared modes, adapting to the differences in resource measurement units between the two modes.

[0054] Optionally, if all verifications pass, then in the target cluster, an attempt is made to filter out target nodes that meet the specifications of the currently traversed object, including: Obtain the GPU type whitelist of the target resource pool; Determine whether the GPU type in the currently traversed specification object is within the GPU type whitelist; If the GPU type is not in the GPU type whitelist, then the target node has not been selected. If the GPU type is within the GPU type whitelist and the currently traversed specification object is an exclusive specification object, then try to filter out the target node that meets the exclusive specification object in the target cluster. If the GPU type is in the GPU type whitelist and the specification object currently being traversed is a shared specification object, then the upper limit of single memory allocation pre-configured for the GPU type by the target resource pool is obtained, and it is determined whether the memory capacity in the shared specification object is greater than the upper limit of single memory allocation. If the video memory capacity is greater than the upper limit of a single video memory request, then it is determined that the target node has not been selected. If the video memory capacity is less than or equal to the upper limit of the single video memory request, then an attempt will be made to filter out target nodes that meet the shared specification object in the target cluster.

[0055] Specifically, in multi-tenant shared clusters, quota verification alone is insufficient for granular resource management. Different GPU models have different computing power and applicable scenarios. Enterprises typically want to restrict low-priority tenants from using high-end GPUs or prevent individual shared tasks from excessively consuming the memory of a particular GPU model, leading to fragmentation. To address this, this embodiment adds heterogeneous resource access control and hard-limit memory verification after quota verification passes but before actual node screening, further strengthening multi-tenant isolation capabilities.

[0056] The GPU type whitelist is a list of GPU models that are allowed to be used within a resource pool, pre-configured by the resource pool administrator. For example, if the whitelist for department A is ["A10", "3090"], then its tenants can only apply for these two types of GPUs, and applications for H100 will be rejected.

[0057] The single-time video memory allocation limit is the maximum video memory capacity that a single shared task can request for a specific GPU model and resource pool configuration. For example, if Department A sets a single-time video memory allocation limit of 16GB for the 3090 card, then any shared task requesting 3090 video memory cannot exceed 16GB to prevent a single card from being over-segmented.

[0058] Existing technologies lack fine-grained access control over heterogeneous GPU models, allowing low-priority tenants to potentially occupy high-end GPUs through shared mode, impacting the computing power supply for core tasks. Excessive memory requests from a single shared task may fragment a large-capacity GPU, causing subsequent large tasks to fail due to insufficient contiguous memory. Different resource pools may have different hardware usage strategies, requiring support for configuring allowed GPU models and single-use memory limits per pool. This embodiment allows resource pools to be configured with allowed GPU models based on business needs, ensuring that high-end computing power is used only for core tasks, while low-priority tasks are automatically restricted to low-cost GPUs. By setting a single-use memory request limit, it prevents a single shared task from fragmenting a large-capacity GPU into too many small pieces, protecting the contiguous memory needs of future large tasks. Different resource pools can be independently configured with whitelists and limits, meeting the differentiated management strategies of multiple departments within an enterprise and avoiding resource contention.

[0059] Optionally, the step of traversing the parsed specification objects in descending order of priority and attempting to filter out target nodes that satisfy the currently traversed specification objects in the target cluster where GPUs are deployed includes: The system attempts to filter candidate nodes in the target cluster that satisfy the specifications of the currently traversed object. Each candidate node satisfies a preset hard constraint. If the currently traversed specification object is an exclusive specification object, the hard constraint is that the candidate node has at least one GPU card that matches the GPU type in the exclusive specification object and is in an idle state. If the currently traversed specification object is a shared specification object, the hard constraint is that the candidate node has at least one GPU card that matches the GPU type in the shared specification object, and the remaining video memory capacity of the GPU card is greater than or equal to the video memory capacity in the shared specification object. If no candidate nodes are found, then the target node has not been found. If a candidate node is selected, then that candidate node is taken as the target node; If multiple candidate nodes are selected, one candidate node is chosen from among them as the target node.

[0060] Specifically, an idle state means that the GPU card is not occupied by any container, and all resources of the entire card are available. It should be noted that each candidate node also satisfies preset soft constraints. If the currently traversed specification object is an exclusive specification object, the soft constraint prioritizes GPU cards under the same NUMA node or the same PCIe switch. This is because GPUs located under the same NUMA node or PCIe switch have higher communication bandwidth and lower latency, and allocating tasks to such nodes can significantly improve the performance of scenarios such as multi-GPU parallel training. If the currently traversed specification object is a shared specification object, the soft constraint prioritizes nodes with partially occupied GPU cards. This is because scheduling shared tasks to GPU cards with existing fragmentation can fill memory gaps, reduce memory fragmentation, and thus reserve more complete free cards for subsequent tasks requiring large blocks of contiguous memory.

[0061] Optionally, if multiple candidate nodes are selected, selecting one candidate node from the multiple candidate nodes as the target node includes: If the specification object being traversed is an exclusive specification object, then determine the total number of GPUs belonging to the GPU type in the exclusive specification object in each candidate node, and determine the number of GPUs belonging to the GPU type and in a non-idle state in each candidate node. Calculate the ratio of the number of GPUs in a non-idle state to the total number of GPUs in each candidate node to obtain the load factor of each candidate node. The candidate node with the highest load factor is determined as the target node.

[0062] Specifically, for each candidate node, the total number of GPUs refers to the total number of GPU cards on that node that match the GPU type in the current exclusive specification object. The number of non-idle GPUs refers to the number of GPU cards of that type on that node that are already occupied by other tasks. A higher load factor indicates a higher proportion of that type of GPU being used on that node, meaning a heavier node load.

[0063] This embodiment selects the node with the highest load factor, rather than the lowest, to prioritize scheduling new tasks to nodes with a certain load. This avoids idle nodes remaining idle for a long time while hot nodes become overloaded, resulting in a more even distribution of load across the cluster. Nodes with high load factors mean they have fewer idle cards. Continuing to schedule tasks to these nodes preserves completely idle nodes, reserving resources for large tasks that may require a large number of continuously idle cards in the future. Furthermore, if the node with the lowest load factor (i.e., the most idle) is always selected, all tasks will be piled onto idle nodes, quickly filling them up while nodes that were originally loaded remain idle, thus failing to achieve true load balancing.

[0064] Optionally, if multiple candidate nodes are selected, selecting one candidate node from the multiple candidate nodes as the target node includes: If the current specification object being traversed is a shared specification object, then determine the total number of GPU types contained in all the parsed specification objects, and determine the priority position of the shared specification object among all the parsed specification objects. Based on the total number of GPU types and the priority ranking, the generational affinity score of the GPU type is determined; wherein, the generational affinity score is negatively correlated with the priority ranking and positively correlated with the total number of GPU types; Determine the total video memory capacity and remaining video memory capacity of all GPU cards belonging to this GPU type in each candidate node; Calculate the ratio of the remaining video memory capacity to the total video memory capacity in each candidate node to obtain the video memory fragmentation rate score for each candidate node. The intergenerational affinity score and memory fragmentation rate score of each candidate node are weighted and summed to obtain the comprehensive score of each candidate node. The candidate node with the highest overall score is determined as the target node.

[0065] Specifically, priority rank refers to the current specification object's position within all specification objects. Higher priority corresponds to a lower rank. For example, in the above declaration, H100 ranks 1, A800 ranks 2, and A10 ranks 3. Generational affinity score is inversely proportional to priority rank and directly proportional to the total number of GPU types. For instance, the generational affinity score is calculated as follows: calculate the difference between the total number of GPU types and the priority rank, and then calculate the ratio of this difference to the total number of GPU types. This ratio serves as the generational affinity score for the GPU types within the shared specification object. The generational affinity score reflects the user's preference order for different GPU models. By converting the priorities in the user's declaration into numerical scores, the scheduler can prioritize the user's preferred GPU model, only using a secondary model when hardware conditions cannot meet the requirements.

[0066] The memory fragmentation score is used to evaluate the memory usage efficiency and fragmentation level of this type of GPU on candidate nodes. A higher score indicates more contiguous memory space and less fragmentation on the node, which is beneficial for scheduling large tasks later. This score is designed to guide the scheduler to prioritize two types of nodes: Nodes with ample remaining video memory: These nodes have a large amount of remaining video memory for this type of GPU card, which can provide sufficient memory space for the current task, while reserving a lot of continuous resources for future large tasks.

[0067] A node that perfectly fills a memory gap: If a node has a GPU card with remaining memory that is exactly equal to the task's requirements, assigning the task to that card can completely fill a memory gap, thereby eliminating fragmentation and improving overall memory utilization.

[0068] When a task can just fill a gap, the remaining memory ratio of the node containing the gap may not be particularly high. However, if that node is the only candidate node that can provide such a match, its remaining memory ratio may still be higher than other nodes that do not meet the criteria at all, thus still having a chance to be selected. Therefore, the score based on the remaining memory ratio can, to some extent, take into account the preference for filling gaps. In a more refined implementation, additional points can be added based on the gap matching degree, but in this embodiment, the score based on the remaining memory ratio can effectively guide the scheduler to prioritize nodes that are conducive to reducing fragmentation.

[0069] Obtain a first weighted value for the intergenerational affinity score and a second weighted value for the memory fragmentation rate score of a candidate node. Calculate a weighted sum of the intergenerational affinity score and the memory fragmentation rate score based on the first and second weighted values, which serves as the comprehensive score for the candidate node. Preferably, the first weighted value is greater than the second weighted value.

[0070] Optionally, the target cluster includes multiple resource pools, each resource pool includes at least one tenant, each tenant includes multiple Pods, and each resource pool pre-sets a reserved requirement for each GPU type, the reserved requirement being used to indicate the number of GPUs that the resource pool wants to reserve for the corresponding GPU type; Each GPU type corresponds to a global reserved water level, which is the maximum value among all resource pools for the reserved demand of the corresponding GPU type; The GPU resources of the target cluster are pre-divided into a global buffer pool and a public resource pool. Both the global buffer pool and the public resource pool include multiple sub-pools. Each sub-pool corresponds to a GPU type. Each sub-pool includes GPU cards of the GPU type corresponding to that sub-pool, located on at least one node. The number of GPUs contained in each sub-pool of the global buffer pool is equal to the global reserved water level for the GPU type corresponding to each sub-pool. The number of GPUs contained in each sub-pool of the public resource pool is equal to the difference between the number of GPUs of the GPU type corresponding to each sub-pool in the target cluster and the global reserved water level.

[0071] Optionally, the step of traversing the parsed specification objects in descending order of priority and attempting to filter out target nodes that satisfy the currently traversed specification objects in the target cluster where GPUs are deployed includes: Determine the task type corresponding to the target Pod; If the task type is a reserved task used to represent core business, then the target node that meets the specification of the currently traversed object will be selected from the global buffer pool. If the task type is a common task used to represent non-core business, then the target node that meets the specifications of the currently traversed object will be selected from the public resource pool.

[0072] It's important to note that traditional resource reservation schemes typically use a static accumulation method to determine the reservation level: the reservation demands of each resource pool are directly summed, and the corresponding number of physical resources are permanently locked as reserved resources. The drawbacks of this approach are: it assumes that sudden peaks in all resource pools occur simultaneously, resulting in a locked amount of resources far exceeding the actual concurrent demand; a large number of reserved resources remain locked but unused during off-peak periods, rendering them unusable for ordinary tasks and causing significant resource waste; and it lacks differentiated reservation management for different GPU models, making it difficult to address the varying management needs of scarce and abundant resources in heterogeneous clusters.

[0073] In multi-tenant shared clusters, ensuring the supply of core business resources while avoiding resource waste caused by static reservation is one of the key issues that this solution needs to address. To this end, this invention proposes a dynamic reservation mechanism based on statistical reuse. By calculating the global reservation level, physical resources are divided into a global buffer pool and a public resource pool, achieving the resource management goal of logical exclusivity and physical reuse.

[0074] Specifically, the reserved demand is a pre-configured value for each resource pool for each GPU type, representing the number of GPU cards that the resource pool expects the system to guarantee for it. This value is set by the administrator based on factors such as business importance, historical peak data, and SLA requirements; it is the guarantee target for the resource pool, not a usage limit. The global reserved water level is the maximum value of the reserved demand for that type across all resource pools, taken as the global reserved water level for that type. This water level is the core parameter of the dynamic reservation mechanism, used to determine the capacity of the global buffer pool. The global buffer pool is a logical resource set composed of physical GPU cards, with a capacity equal to the global reserved water level for each GPU type. GPU cards in this pool are only allowed to be used by core tasks with reserved tags, ensuring resource availability even in the event of a sudden surge in any single resource pool. The public resource pool is also a logical resource set composed of physical GPU cards, with a capacity equal to the total number of GPU cards of each GPU type in the cluster minus the global reserved water level. GPU cards in this pool are open to all ordinary tasks, achieving full utilization of resources. Sub-pools refer to the global buffer pool and public resource pool, which are further divided into multiple sub-pools according to GPU type. Each sub-pool corresponds to a GPU type and contains GPU cards of that type distributed on at least one node.

[0075] For example, suppose a cluster contains three resource pools, A, B, and C, with the following reservation requirements for H100 and A800 GPUs: resource pool A reserves 8 and 5 GPUs respectively, resource pool B reserves 4 and 10 GPUs respectively, and resource pool C reserves 2 and 3 GPUs respectively. Then, for the H100 type, the global reservation level is 8 GPUs; for the A800 type, the global reservation level is 10 GPUs.

[0076] The global buffer pool and the public resource pool are complementary, together constituting all the physical resources of the GPU type in the target cluster. A GPU card cannot belong to both pools simultaneously.

[0077] In this embodiment, the global buffer pool has sufficient capacity to meet the sudden demands of any single resource pool, ensuring that core tasks are still guaranteed even when resources are scarce. Based on the principle of statistical reuse, only the maximum value of the reserved demand for each resource pool is locked, rather than a simple summation, releasing a large amount of idle reserved resources for use by ordinary tasks. For example, in the example, the reserved resources for H100 are reduced from 14 to 8, freeing up 6 expensive high-performance cards to enter the public pool, improving resource utilization by 43%. Water levels are calculated independently for different GPU types, adapting to the differentiated management needs of scarce and abundant resources in heterogeneous clusters.

[0078] It should be noted that when there are multiple targets to be processed, this embodiment uses one target as an example to explain the specific processing procedure in detail. The processing methods for other targets are the same as those for the target in this example. The targets to be processed can be candidate nodes, Pods, specification objects, etc.

[0079] The following is combined Figures 2 to 5 The technical solution of this application is explained with examples.

[0080] like Figure 2 As shown, the system is built on the Kubernetes container orchestration platform. While retaining the native scheduler, it integrates a hybrid scheduling core module through the Scheduler Extender mechanism. This module includes a configuration parsing engine, a multi-tenant quota management unit, a heterogeneous resource scoring unit, and a dynamic watermark calculation unit. The system receives Pod creation requests submitted by users through the API Server, parses GPU resource declarations from Pod annotations, obtains cluster node resource information, tenant quotas, and resource pool reservation configurations by accessing ETCD, and finally, the Scheduler Extender performs node filtering and optimization decisions, writing the binding results back to the API Server. Figure 2 It also demonstrates the logical division between the global buffer pool and the public resource pool, as well as their relationship with physical nodes, reflecting the overall technical architecture of the present invention.

[0081] like Figure 3As shown, the process begins with obtaining the GPU resource declaration of the target Pod, followed by a configuration parsing step, where multiple configuration items in the declaration are parsed into specification objects according to priority. Next, multi-tenant quota verification is performed, including tenant-level and resource pool-level checks. After quota verification passes, access control checks are performed, including GPU type whitelisting and single-use memory limit verification in shared mode. Then, the node selection phase begins: candidate nodes are first selected based on hard constraints. If multiple candidate nodes are selected, full load rate optimization or comprehensive score optimization is performed according to the specification object type (exclusive / shared) to determine the final target node. If the current specification object selection fails, it automatically downgrades to try the next priority specification object until success or traversal is complete. The flowchart fully illustrates the complete scheduling decision chain from resource declaration to node binding.

[0082] like Figure 4 As shown, the system comprises multiple resource pools (such as Resource Pool A and Resource Pool B), each corresponding to an organizational unit. Each resource pool contains multiple tenants, each corresponding to a team or project. Each tenant can submit multiple Pods and has an independent GPU quota. Quota limits are also set at the resource pool level to control the total resources available to the entire department. Furthermore, each resource pool pre-configures reserved demand for each GPU type as input to the global reserved water level. Figure 4 The hierarchical relationship of "resource pool-tenant-pod" and the attachment position of quota and reserved configuration are clearly displayed in a tree structure, which reflects the design concept of multi-tenant fine-grained management and control of this invention.

[0083] Figure 5 This diagram illustrates the unified declarative syntax parsing process provided by an embodiment of the present invention. The left side of the diagram shows a user configuration example: “H100:2, A800 / 24, A10 / 16”, a string carried in a Pod annotation. The middle of the diagram shows the parser's word segmentation and recognition process: configuration items are separated by commas, a pattern identifier is identified for each configuration item, and the GPU model and resource quantity values ​​are separated using the identifier as a boundary. The right side of the diagram shows the parsing result: a queue of specification objects is generated, ordered by priority. Exclusive specification objects contain the GPU type and number of GPUs, while shared specification objects contain the GPU type and memory capacity. The queue order decreases in priority from left to right. This diagram visually illustrates the conversion process from user declarations to structured specification objects, demonstrating the simplicity and parsability of the unified syntax design of this invention.

[0084] Example 2 This invention provides a GPU resource scheduling device, such as... Figure 6 As shown, the GPU resource scheduling device 60 specifically includes the following components: The acquisition module 601 is used to acquire the GPU resource declaration carried in the target Pod; wherein, the GPU resource declaration includes multiple GPU resource configuration items arranged in descending order of priority; The parsing module 602 is used to parse each GPU resource configuration item in sequence to obtain the specification object corresponding to each GPU resource configuration item; wherein, the exclusive specification object used to represent the exclusive GPU includes the GPU type and the number of GPUs of that type, and the shared specification object used to represent the shared GPU includes the GPU type and the video memory capacity. The filtering module 603 is used to traverse the parsed specification objects in descending order of priority and attempt to filter out target nodes that meet the specification objects currently being traversed in the target cluster where GPUs are deployed. The scheduling module 604 is used to bind the target Pod to the target node when the target node is selected, and to allocate GPU resources to the target Pod according to the specification object currently traversed. The determination module 605 is used to continue traversing the parsed specification objects if the target node is not selected, until the target node is selected or the traversal is completed.

[0085] Optionally, the parsing module is specifically used for: Parse the mode identifier, GPU model, and resource quantity values ​​carried in the GPU resource configuration item; If the pattern identifier is an exclusive identifier used to characterize an exclusive GPU, then the data format associated with the exclusive identifier is obtained, the GPU model is used as the GPU type in the data format, and the resource quantity value is used as the number of GPUs in the data format to construct the exclusive specification object. If the pattern identifier is a shared identifier used to characterize a shared GPU, then the data format associated with the shared identifier is obtained, the GPU model is used as the GPU type in the data format, and the resource quantity value is used as the video memory capacity in the data format to construct the shared specification object.

[0086] Optionally, the filtering module is specifically used for: Obtain the GPU quota of the target tenant to which the target Pod belongs and the GPU quota of the target resource pool to which the target tenant belongs; wherein, the GPU quota includes the upper limit of the number of GPUs and the upper limit of the video memory capacity; If the current traversed specification object is an exclusive specification object, then check whether the sum of the number of GPUs used by the target tenant and the number of GPUs in the exclusive specification object exceeds the upper limit of the number of GPUs of the target tenant, and check whether the sum of the number of GPUs used by all tenants in the target resource pool and the number of GPUs in the exclusive specification object exceeds the upper limit of the number of GPUs of the target resource pool. If the currently traversed specification object is a shared specification object, then check whether the sum of the video memory capacity used by the target tenant and the video memory capacity in the shared specification object exceeds the upper limit of the video memory capacity of the target tenant, and check whether the sum of the video memory capacity used by all tenants in the target resource pool and the video memory capacity in the shared specification object exceeds the upper limit of the video memory capacity of the target resource pool. If all verifications pass, then try to filter out target nodes in the target cluster that meet the specifications of the currently traversed object; If any verification fails, it is determined that the target node has not been selected.

[0087] Optionally, when the filtering module attempts to filter out target nodes in the target cluster that meet the specifications of the currently traversed object if all verifications pass, it is specifically used for: Obtain the GPU type whitelist of the target resource pool; Determine whether the GPU type in the currently traversed specification object is within the GPU type whitelist; If the GPU type is not in the GPU type whitelist, then the target node has not been selected. If the GPU type is within the GPU type whitelist and the currently traversed specification object is an exclusive specification object, then try to filter out the target node that meets the exclusive specification object in the target cluster. If the GPU type is in the GPU type whitelist and the specification object currently being traversed is a shared specification object, then the upper limit of single memory allocation pre-configured for the GPU type by the target resource pool is obtained, and it is determined whether the memory capacity in the shared specification object is greater than the upper limit of single memory allocation. If the video memory capacity is greater than the upper limit of a single video memory request, then it is determined that the target node has not been selected. If the video memory capacity is less than or equal to the upper limit of the single video memory request, then an attempt will be made to filter out target nodes that meet the shared specification object in the target cluster.

[0088] Optionally, the filtering module is specifically used for: The system attempts to filter candidate nodes in the target cluster that satisfy the specifications of the currently traversed object. Each candidate node satisfies a preset hard constraint. If the currently traversed specification object is an exclusive specification object, the hard constraint is that the candidate node has at least one GPU card that matches the GPU type in the exclusive specification object and is in an idle state. If the currently traversed specification object is a shared specification object, the hard constraint is that the candidate node has at least one GPU card that matches the GPU type in the shared specification object, and the remaining video memory capacity of the GPU card is greater than or equal to the video memory capacity in the shared specification object. If no candidate nodes are found, then the target node has not been found. If a candidate node is selected, then that candidate node is taken as the target node; If multiple candidate nodes are selected, one candidate node is chosen from among them as the target node.

[0089] Optionally, when the filtering module performs the step of selecting one candidate node from multiple candidate nodes as the target node if multiple candidate nodes are selected, it is specifically used for: If the specification object being traversed is an exclusive specification object, then determine the total number of GPUs belonging to the GPU type in the exclusive specification object in each candidate node, and determine the number of GPUs belonging to the GPU type and in a non-idle state in each candidate node. Calculate the ratio of the number of GPUs in a non-idle state to the total number of GPUs in each candidate node to obtain the load factor of each candidate node. The candidate node with the highest load factor is determined as the target node.

[0090] Optionally, when the filtering module performs the step of selecting one candidate node from multiple candidate nodes as the target node if multiple candidate nodes are selected, it is further configured to: If the current specification object being traversed is a shared specification object, then determine the total number of GPU types contained in all the parsed specification objects, and determine the priority position of the shared specification object among all the parsed specification objects. Based on the total number of GPU types and the priority ranking, the generational affinity score of the GPU type is determined; wherein, the generational affinity score is negatively correlated with the priority ranking and positively correlated with the total number of GPU types; Determine the total video memory capacity and remaining video memory capacity of all GPU cards belonging to this GPU type in each candidate node; Calculate the ratio of the remaining video memory capacity to the total video memory capacity in each candidate node to obtain the video memory fragmentation rate score for each candidate node. The intergenerational affinity score and memory fragmentation rate score of each candidate node are weighted and summed to obtain the comprehensive score of each candidate node. The candidate node with the highest overall score is determined as the target node.

[0091] Optionally, the target cluster includes multiple resource pools, each resource pool includes at least one tenant, each tenant includes multiple Pods, and each resource pool pre-sets a reserved requirement for each GPU type, the reserved requirement being used to indicate the number of GPUs that the resource pool wants to reserve for the corresponding GPU type; Each GPU type corresponds to a global reserved water level, which is the maximum value among all resource pools for the reserved demand of the corresponding GPU type; The GPU resources of the target cluster are pre-divided into a global buffer pool and a public resource pool. Both the global buffer pool and the public resource pool include multiple sub-pools. Each sub-pool corresponds to a GPU type. Each sub-pool includes GPU cards of the GPU type corresponding to that sub-pool, located on at least one node. The number of GPUs contained in each sub-pool of the global buffer pool is equal to the global reserved water level for the GPU type corresponding to each sub-pool. The number of GPUs contained in each sub-pool of the public resource pool is equal to the difference between the number of GPUs of the GPU type corresponding to each sub-pool in the target cluster and the global reserved water level.

[0092] Optionally, the filtering module is specifically used for: Determine the task type corresponding to the target Pod; If the task type is a reserved task used to represent core business, then the target node that meets the specification of the currently traversed object will be selected from the global buffer pool. If the task type is a common task used to represent non-core business, then the target node that meets the specifications of the currently traversed object will be selected from the public resource pool.

[0093] Example 3 This embodiment also provides a computer device, such as a smartphone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including a standalone server or a server cluster composed of multiple servers), etc., capable of executing programs. Figure 7 As shown, the computer device 70 in this embodiment includes, but is not limited to, a memory 701 and a processor 702 that are communicatively connected to each other via a system bus. It should be noted that... Figure 7 Only a computer device 70 with components 701-702 is shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0094] In this embodiment, the memory 701 (i.e., the readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 701 may be an internal storage unit of the computer device 70, such as the hard disk or memory of the computer device 70. In other embodiments, the memory 701 may also be an external storage device of the computer device 70, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 70. Of course, the memory 701 may include both the internal storage unit and its external storage device of the computer device 70. In this embodiment, the memory 701 is typically used to store the operating system and various application software installed on the computer device 70. In addition, the memory 701 may also be used to temporarily store various types of data that have been output or will be output.

[0095] In some embodiments, processor 702 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. This processor 702 is typically used to control the overall operation of the computer device 70.

[0096] Specifically, in this embodiment, the processor 702 is used to execute the program of the GPU resource scheduling method stored in the memory 701.

[0097] For a detailed description of the above method steps, please refer to Example 1. This example will not be repeated here.

[0098] Example 4 This embodiment also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., which stores a computer program. When the computer program is executed by a processor, it is used to implement the steps of the GPU resource scheduling method.

[0099] For a detailed description of the above method steps, please refer to Example 1. This example will not be repeated here.

[0100] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0101] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0103] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A GPU resource scheduling method, characterized in that, include: Obtain the GPU resource declaration carried in the target Pod; wherein, the GPU resource declaration includes multiple GPU resource configuration items arranged in descending order of priority; Each GPU resource configuration item is parsed sequentially to obtain the specification object corresponding to each GPU resource configuration item; among them, the exclusive specification object used to represent the exclusive GPU includes the GPU type and the number of GPUs of that type, and the shared specification object used to represent the shared GPU includes the GPU type and the video memory capacity. The parsed specification objects are traversed in descending order of priority, and target nodes that meet the specification objects currently being traversed are selected from the target clusters where GPUs are deployed. If the target node is selected, the target Pod is bound to the target node, and GPU resources are allocated to the target Pod according to the specification object currently traversed. If the target node is not found, continue traversing the parsed specification objects until the target node is found or the traversal is complete.

2. The GPU resource scheduling method according to claim 1, characterized in that, The step of sequentially parsing each GPU resource configuration item to obtain the specification object corresponding to each GPU resource configuration item includes: Parse the mode identifier, GPU model, and resource quantity values ​​carried in the GPU resource configuration item; If the pattern identifier is an exclusive identifier used to characterize an exclusive GPU, then the data format associated with the exclusive identifier is obtained, the GPU model is used as the GPU type in the data format, and the resource quantity value is used as the number of GPUs in the data format to construct the exclusive specification object. If the pattern identifier is a shared identifier used to characterize a shared GPU, then the data format associated with the shared identifier is obtained, the GPU model is used as the GPU type in the data format, and the resource quantity value is used as the video memory capacity in the data format to construct the shared specification object.

3. The GPU resource scheduling method according to claim 1, characterized in that, The process of traversing the parsed specification objects in descending order of priority, and attempting to filter out target nodes that satisfy the currently traversed specification objects in the target cluster with GPUs, includes: Obtain the GPU quota of the target tenant to which the target Pod belongs and the GPU quota of the target resource pool to which the target tenant belongs; wherein, the GPU quota includes the upper limit of the number of GPUs and the upper limit of the video memory capacity; If the current traversed specification object is an exclusive specification object, then check whether the sum of the number of GPUs used by the target tenant and the number of GPUs in the exclusive specification object exceeds the upper limit of the number of GPUs of the target tenant, and check whether the sum of the number of GPUs used by all tenants in the target resource pool and the number of GPUs in the exclusive specification object exceeds the upper limit of the number of GPUs of the target resource pool. If the currently traversed specification object is a shared specification object, then check whether the sum of the video memory capacity used by the target tenant and the video memory capacity in the shared specification object exceeds the upper limit of the video memory capacity of the target tenant, and check whether the sum of the video memory capacity used by all tenants in the target resource pool and the video memory capacity in the shared specification object exceeds the upper limit of the video memory capacity of the target resource pool. If all verifications pass, then try to filter out target nodes in the target cluster that meet the specifications of the currently traversed object; If any verification fails, it is determined that the target node has not been selected.

4. The GPU resource scheduling method according to claim 3, characterized in that, If all verifications pass, then in the target cluster, an attempt is made to filter out target nodes that meet the specifications of the currently traversed object, including: Obtain the GPU type whitelist of the target resource pool; Determine whether the GPU type in the currently traversed specification object is within the GPU type whitelist; If the GPU type is not in the GPU type whitelist, then the target node has not been selected. If the GPU type is within the GPU type whitelist and the currently traversed specification object is an exclusive specification object, then try to filter out the target node that meets the exclusive specification object in the target cluster. If the GPU type is in the GPU type whitelist and the specification object currently being traversed is a shared specification object, then the upper limit of single memory allocation pre-configured for the GPU type by the target resource pool is obtained, and it is determined whether the memory capacity in the shared specification object is greater than the upper limit of single memory allocation. If the video memory capacity is greater than the upper limit of a single video memory request, then it is determined that the target node has not been selected. If the video memory capacity is less than or equal to the upper limit of the single video memory request, then an attempt will be made to filter out target nodes that meet the shared specification object in the target cluster.

5. The GPU resource scheduling method according to claim 1, characterized in that, The process of traversing the parsed specification objects in descending order of priority, and attempting to filter out target nodes that satisfy the currently traversed specification objects in the target cluster with GPUs, includes: The system attempts to filter candidate nodes in the target cluster that satisfy the specifications of the currently traversed object. Each candidate node satisfies a preset hard constraint. If the currently traversed specification object is an exclusive specification object, the hard constraint is that the candidate node has at least one GPU card that matches the GPU type in the exclusive specification object and is in an idle state. If the currently traversed specification object is a shared specification object, the hard constraint is that the candidate node has at least one GPU card that matches the GPU type in the shared specification object, and the remaining video memory capacity of the GPU card is greater than or equal to the video memory capacity in the shared specification object. If no candidate nodes are found, then the target node has not been found. If a candidate node is selected, then that candidate node is taken as the target node; If multiple candidate nodes are selected, one candidate node is chosen from among them as the target node.

6. The GPU resource scheduling method according to claim 5, characterized in that, If multiple candidate nodes are selected, then selecting one candidate node from the multiple candidate nodes as the target node includes: If the specification object being traversed is an exclusive specification object, then determine the total number of GPUs belonging to the GPU type in the exclusive specification object in each candidate node, and determine the number of GPUs belonging to the GPU type and in a non-idle state in each candidate node. Calculate the ratio of the number of GPUs in a non-idle state to the total number of GPUs in each candidate node to obtain the load factor of each candidate node. The candidate node with the highest load factor is determined as the target node.

7. The GPU resource scheduling method according to claim 5, characterized in that, If multiple candidate nodes are selected, then selecting one candidate node from the multiple candidate nodes as the target node includes: If the current specification object being traversed is a shared specification object, then determine the total number of GPU types contained in all the parsed specification objects, and determine the priority position of the shared specification object among all the parsed specification objects. Based on the total number of GPU types and the priority ranking, the generational affinity score of the GPU type is determined; wherein, the generational affinity score is negatively correlated with the priority ranking and positively correlated with the total number of GPU types; Determine the total video memory capacity and remaining video memory capacity of all GPU cards belonging to this GPU type in each candidate node; Calculate the ratio of the remaining video memory capacity to the total video memory capacity in each candidate node to obtain the video memory fragmentation rate score for each candidate node. The intergenerational affinity score and memory fragmentation rate score of each candidate node are weighted and summed to obtain the comprehensive score of each candidate node. The candidate node with the highest overall score is determined as the target node.

8. The GPU resource scheduling method according to claim 1, characterized in that, The target cluster includes multiple resource pools, each resource pool includes at least one tenant, each tenant includes multiple Pods, and each resource pool pre-sets a reserved requirement for each GPU type. The reserved requirement is used to indicate the number of GPUs that the resource pool wants to reserve for the corresponding GPU type. Each GPU type corresponds to a global reserved water level, which is the maximum value among all resource pools for the reserved demand of the corresponding GPU type; The GPU resources of the target cluster are pre-divided into a global buffer pool and a public resource pool. Both the global buffer pool and the public resource pool include multiple sub-pools. Each sub-pool corresponds to a GPU type. Each sub-pool includes GPU cards of the GPU type corresponding to that sub-pool, located on at least one node. The number of GPUs contained in each sub-pool of the global buffer pool is equal to the global reserved water level for the GPU type corresponding to each sub-pool. The number of GPUs contained in each sub-pool of the public resource pool is equal to the difference between the number of GPUs of the GPU type corresponding to each sub-pool in the target cluster and the global reserved water level.

9. The GPU resource scheduling method according to claim 8, characterized in that, The process of traversing the parsed specification objects in descending order of priority, and attempting to filter out target nodes that satisfy the currently traversed specification objects in the target cluster with GPUs, includes: Determine the task type corresponding to the target Pod; If the task type is a reserved task used to represent core business, then the target node that meets the specification of the currently traversed object will be selected from the global buffer pool. If the task type is a common task used to represent non-core business, then the target node that meets the specifications of the currently traversed object will be selected from the public resource pool.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it is used to implement the steps of the method according to any one of claims 1 to 9.

Images