GPU (Graphics Processing Unit) resource intelligent allocation method, device and equipment for AI (Artificial Intelligence) push training platform

Abstract

The embodiment of the invention relates to the technical field of cloud computing, in particular to a GPU resource intelligent allocation method and device for an AI push training platform and electronic equipment. The method comprises the following steps: regularly collecting GPU multi-dimensional operation state data of each GPU node in an AI push training platform, and classifying and storing the GPU multi-dimensional operation state data to generate a GPU resource state library; monitoring a pod creation event, identifying a task type corresponding to a new pod, and formulating a matched GPU scheduling strategy; judging whether the available resources meet the requirements of a GPU scheduling strategy or not according to the GPU resource state library; if yes, executing a scheduling distribution operation corresponding to the GPU scheduling strategy; and if the available resources do not meet the requirements of the GPU scheduling strategy, preempting the online development task resources meeting the preset conditions, and executing the step of judging whether the available resources meet the requirements of the GPU scheduling strategy according to the GPU resource state library. According to the method, accurate identification and differentiated scheduling of the hybrid load can be realized, and the GPU resource utilization rate is improved.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a method, apparatus and electronic device for intelligent allocation of GPU resources for AI training platforms. Background Technology

[0002] AI training platforms are one-stop technology carriers that support the entire process of artificial intelligence model development and training. Based on container orchestration and management tools (K8S, Kubernetes), they integrate core hardware resources such as GPUs (graphics processing units) and various development and training tools, providing R&D personnel with an efficient environment that allows them to focus on model algorithm optimization without having to worry about underlying resource management. They are widely used.

[0003] GPUs serve as the core computing power support for AI training platforms, and their allocation efficiency directly affects the progress of AI research and development and the cost of resource utilization. Existing GPU scheduling solutions in container orchestration management tools (Kubernetes) primarily employ an allocation mechanism based on resource requests and limits. They schedule Pods (the smallest deployable computing unit) to worker nodes that meet resource requirements through First-Fit algorithms or random strategies. Some solutions integrate batch schedulers (such as Volcano) to support features such as batch scheduling and priority queues, meeting the basic scheduling requirements of distributed training.

[0004] However, existing technologies have several significant shortcomings: First, the scheduler makes decisions based solely on the amount of resource requests, lacking the ability to perceive and distinguish mixed load characteristics. This results in GPUs often having an actual utilization rate of less than 20% in online development scenarios, yet they occupy resources for extended periods, causing offline training tasks to wait for long periods due to insufficient resources. Second, GPU resource utilization is low, with a serious problem of resources being occupied but not used, resulting in significant waste. Third, distributed training task scheduling is inefficient and prone to resource deadlock.

[0005] Therefore, how to solve the problems of resource hoarding without use, imbalanced allocation, and scheduling deadlock in AI training platforms is an urgent issue for those skilled in the art. Summary of the Invention

[0006] The purpose of this application is to provide at least one method, device, and electronic device for intelligent allocation of GPU resources for AI training platforms, which can achieve accurate identification and differentiated scheduling of mixed loads and improve GPU resource utilization.

[0007] To address the aforementioned technical problems, at least one embodiment of this application provides a method for intelligent allocation of GPU resources for an AI training platform, comprising: Regularly collect multi-dimensional GPU operating status data from each GPU node in the AI ​​training platform, classify and store the data, and generate a GPU resource status database. Listen for pod creation events. When a pod creation event is detected, identify the task type corresponding to the new pod. The task types include: online development tasks and offline training tasks. Develop a GPU scheduling strategy that matches the task type; The system determines whether the available resources meet the requirements of the GPU scheduling policy based on the GPU resource status database. If they do, the system performs the scheduling and allocation operation corresponding to the GPU scheduling policy. If they do not meet the requirements, the system preempts online development task resources that meet the preset conditions and then performs the step of determining whether the available resources meet the requirements of the GPU scheduling policy based on the GPU resource status database.

[0008] In one embodiment, formulating a GPU scheduling policy that matches the task type includes: If the task type is an online development task, a centralized deployment strategy is generated; the centralized deployment strategy includes: prioritizing allocation to nodes that have already run similar tasks and have remaining GPUs, allowing GPU resource over-allocation, and configuring resources by setting resource limits without setting resource requests; If the task type is an offline training task, a dedicated exclusive deployment strategy is generated; the dedicated exclusive deployment strategy includes: prioritizing allocation to GPU nodes that meet preset idle conditions or only run similar tasks, configuring resources while setting resource requests and resource limits.

[0009] In one embodiment, if the GPU scheduling policy is the centralized deployment policy, then the step of executing the scheduling and allocation operation corresponding to the GPU scheduling policy includes: The target nodes are selected according to the centralized deployment strategy, and the GPU usage mode specified by the user is received. If the GPU usage mode is a dedicated full-card mode, the GPU resource configuration parameters of the target node are set to the number of full cards specified by the user; If the GPU usage mode is shared mode, the GPU resource configuration parameters of the target node are set to a preset sharing identifier, and GPU sharing is performed through GPU virtualization technology; If the GPU usage mode is virtual GPU mode, the GPU resource configuration parameters of the target node are set to a virtual GPU share divided according to a preset ratio, and the virtual GPU is allocated and used according to GPU slicing technology.

[0010] In one embodiment, the intelligent GPU resource allocation method for an AI training platform further includes: Determine whether the offline training task is a single-machine offline training task or a distributed offline training task; If it belongs to the distributed offline training task, add batch scheduling annotation configuration to the dedicated deployment strategy; the batch scheduling annotation configuration includes: specifying the batch scheduling enable flag and the minimum available number threshold of the associated pods required by the task, so as to ensure that all associated pods are scheduled at the same time.

[0011] In one embodiment, if the GPU scheduling policy is the dedicated exclusive deployment policy, then the execution of the scheduling allocation operation corresponding to the GPU scheduling policy includes: Determine whether the GPU usage in the exclusive deployment strategy is a single card or multiple cards; If it is a single card, the target node is selected according to the exclusive deployment strategy, and the GPU in the target node that meets the resource requirements is bound to the new pod; If there are multiple GPUs, read the GPU topology association data in the GPU resource status database, enumerate the GPU combinations that meet the task's GPU quantity requirements, score each GPU combination according to the topology connection type, and select the GPU combination with the highest score to bind to the new pod.

[0012] In one embodiment, the preemption of online development task resources that meet preset conditions includes: Filter target pods that meet the preset conditions from the GPU resource status database; Send a preemption reminder and a resource release countdown notification to the user to which the target pod belongs, and wait for a preset response time; If the resources corresponding to the target pod are released within the preset response time, then resource reclamation will be initiated. If the resources corresponding to the target pod are not released within the preset response time, a deletion signal is sent to the target pod and a termination time is set. After the termination time is reached, the process of the target pod is forcibly terminated, and the corresponding preemption information is stored. The preemption information includes: timestamp, information of the preempted pod, user information, reason for preemption, and information of the newly scheduled pod.

[0013] In one embodiment, the step of filtering target pods that meet the preset conditions from the GPU resource status database includes: Pods corresponding to online development tasks are selected from the GPU resource status database as candidate pods; Combining GPU utilization data and idle time records, pods with GPU utilization below a preset utilization threshold and idle time exceeding a preset time threshold are extracted from the candidate pods and used as pre-selected pods. Exclude pseudo-idle pods from the pre-selected pods that are in the initial startup phase, process hibernation, or waiting for input / output, and use them as the initial selection pods; The preemption cost of each initially selected pod is calculated by calling a preset formula, and the pod with the lowest preemption cost is selected as the target pod. The preset formula is obtained by performing a positive correlation weighted calculation on the pod runtime and a negative correlation weighted calculation on the GPU idle time corresponding to the pod.

[0014] In one embodiment, the intelligent GPU resource allocation method for an AI training platform further includes: Idle GPUs in online development tasks are determined based on the multi-dimensional GPU operating status data. Determine whether the idle time of the idle GPU reaches the idle threshold standard corresponding to the online development task; If this is achieved, the idle GPU will be reclaimed in a gradient manner.

[0015] At least one embodiment of this application also provides a GPU resource intelligent allocation device for an AI training platform, comprising: The data acquisition unit is used to periodically collect multi-dimensional running status data of each GPU node in the AI ​​training platform, classify and store it, and generate a GPU resource status library. A pod creation identification unit is used to listen for pod creation events. When a pod creation event is detected, the unit identifies the task type corresponding to the new pod. The task types include online development tasks and offline training tasks. The scheduling strategy formulation unit is used to formulate a GPU scheduling strategy that matches the task type. The real-time resource judgment unit is used to determine whether the available resources meet the requirements of the GPU scheduling policy based on the GPU resource status database; if they meet the requirements, the scheduling allocation unit is triggered; if they do not meet the requirements, the resource preemption unit is triggered. The scheduling and allocation unit is used to execute the scheduling and allocation operation corresponding to the GPU scheduling policy; The resource preemption unit is used to preempt online development task resources that meet preset conditions, and then trigger the real-time resource judgment unit.

[0016] At least one embodiment of this application also provides an electronic device, including: at least one processor; and a memory communicatively connected to said at least one processor; The memory stores instructions that can be executed by the at least one processor, which are then executed by the at least one processor to enable the at least one processor to execute the GPU resource intelligent allocation method for the AI ​​training platform.

[0017] The embodiments of this application provide a method for intelligent GPU resource allocation for an AI training platform. This method periodically collects multi-dimensional operational status data of GPU nodes and stores it in a categorized manner to generate a resource status database, ensuring that the cluster's GPU resource status is real-time and readily available. It also monitors pod creation events, capturing key pod creation events and identifying two core tasks: online development and offline training. This overcomes the limitation of existing technologies that cannot perceive differences in load characteristics, providing a core basis for formulating differentiated scheduling strategies and avoiding resource conflicts caused by indiscriminate treatment of different loads. Furthermore, it addresses the challenges of high resource fluctuations and long-term low utilization in online development tasks, and high-load, stable utilization in offline training tasks, requiring avoidance of resource conflicts. With its inherent interference-free characteristics and customized scheduling logic, it solves the problem of single and rigid scheduling strategies in existing technologies, achieving precise adaptation of scheduling rules and ensuring that resource allocation is highly aligned with task requirements. Finally, it identifies the currently available resources based on the GPU resource status library. If the available resources can meet the requirements of the current GPU scheduling strategy, it directly executes the adapted scheduling to ensure rapid task deployment. If resources are insufficient, it preempts idle resources from eligible online development tasks. This not only solves the resource gap in offline training tasks but also maximizes the reuse of idle resources occupied by online development tasks, avoiding resource waste and task blocking, and balancing resource utilization and task operation stability.

[0018] This method achieves an efficient balance between GPU resource utilization and task operation stability on the AI ​​training platform through a closed-loop mechanism that includes multi-dimensional GPU resource data collection, accurate task type identification, differentiated scheduling strategy formulation, and targeted preemption of idle online development resources. Attached Figure Description

[0019] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0020] Figure 1 This is a flowchart of an embodiment of the present application providing a method for intelligent allocation of GPU resources for an AI training platform; Figure 2 This is a flowchart of another method for intelligent allocation of GPU resources for an AI training platform, provided in one embodiment of this application; Figure 3 This is a schematic diagram of the module division of another GPU resource intelligent allocation device for an AI training platform provided in one embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0022] The following is a detailed description of the implementation details of the GPU resource intelligent allocation method for the AI ​​training platform in this embodiment. The following content is only for the convenience of understanding and is not necessary for implementing this solution.

[0023] Example 1: The specific process of the intelligent GPU resource allocation method for the AI ​​training platform in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 101: Regularly collect multi-dimensional GPU operating status data of each GPU node in the AI ​​training platform, classify and store the data, and generate a GPU resource status database.

[0024] The AI ​​training platform uses a pre-defined data acquisition mechanism to continuously acquire multi-dimensional runtime status data of all GPU nodes in the cluster at fixed time intervals. The data acquisition target is each GPU node, specifically all worker nodes (Nodes) in the AI ​​training platform's Kubernetes cluster that are configured with GPU hardware, excluding ordinary computing nodes that only provide CPU and memory resources. This ensures accurate data acquisition while reducing the workload of data collection and computation. Data acquisition adopts a periodic acquisition mode, continuously performing data acquisition at preset fixed time intervals (e.g., 10 seconds / time) to ensure real-time data acquisition.

[0025] The collected data focuses on multi-dimensional GPU operating status data, including but not limited to: real-time GPU hardware operating data (such as utilization, memory usage, temperature, power consumption), hardware inherent attribute data (such as model), topology association data (such as GPU connection methods, PCIe / NVLink topology relationships, etc.), pod binding and resource configuration data with GPU (such as bound pod name, pod type, resource configuration parameters, etc.), and maintains GPU idle time records as multi-dimensional GPU operating status data.

[0026] The collected multi-dimensional data is classified and stored. For example, static data can be stored in the form of node labels. Static data includes topological association data and hardware inherent attribute data. Dynamic data is stored in a distributed key-value storage system through custom resources. Dynamic data includes: real-time GPU hardware operation data, real-time binding and resource configuration data between pods and GPUs, and real-time GPU idle time records.

[0027] By integrating and aggregating all categorized and stored GPU data to form a unified GPU resource status database, centralized management of cluster GPU resource information is achieved, rather than scattered storage on various nodes. This eliminates the need to query data on each node individually, allowing for quick and direct retrieval of required information directly from the resource database, thereby improving the response speed and accuracy of scheduling decisions.

[0028] Step 102: Listen for pod creation events. When a pod creation event is detected, identify the task type corresponding to the new pod.

[0029] The scheduling decision component of the AI ​​training platform continuously captures pod creation events in the K8S cluster through a preset event listening mechanism. The listening scope covers all pod creation operations related to GPU nodes in the cluster, ensuring that no new tasks that require GPU resource allocation are missed.

[0030] When a new pod creation request is detected, the task type identification process is immediately initiated. Based on preset rules, the workload type corresponding to the new pod is determined, ultimately classifying it as an online development task or an offline training task. This provides a core basis for subsequently formulating differentiated scheduling strategies. A new pod refers to a pod that has just initiated a creation request but has not yet completed GPU resource allocation and node binding.

[0031] The resource requirements and usage characteristics of online development tasks and offline training tasks differ significantly. Online development tasks (such as container instances / Notebooks) exhibit highly volatile resource requirements, alternating between long periods of low load and short periods of high load. GPU utilization is often below 20%. They support multi-task resource sharing, are interactive, have long lifecycles (lasting from days to weeks), have lower stability requirements, and can recover quickly after interruptions. They prefer centralized deployment and flexible configuration (full GPU / shared / virtual GPU). Offline training tasks (such as batch training jobs) have stable and consistently high resource requirements. GPU utilization during training is generally above 70%. They require dedicated resources to avoid interference, operate in batch processing mode, have short lifecycles (resources are released immediately upon completion), and have extremely high stability requirements. Interruptions can lead to data loss or training failure. They prefer independent node deployment and fixed resource configurations and support multi-GPU / distributed topology optimization.

[0032] Existing technologies employ a static allocation method based on resource requests and limits, scheduling nodes through First-Fit or random strategies. This lacks the ability to perceive and differentiate the mixed load characteristics of the two types of tasks. It neither designs over-allocation and resource sharing mechanisms for online development tasks nor meets the exclusive resource and isolation requirements of offline training tasks. As a result, online development tasks occupy resources but do not use them, resulting in a resource utilization rate of less than 20%. Offline training tasks wait for a long time due to resource occupation. The problem of mutual interference between the two types of tasks when they are deployed together is prominent.

[0033] To address this issue and overcome the pain point of indiscriminately treating existing technical workloads, thus providing a basis for developing differentiated scheduling strategies, this method identifies task types. For online development tasks characterized by large resource fluctuations and long-term low utilization, centralized deployment and over-allocation strategies can be triggered upon identification to achieve resource reuse during off-peak periods. For offline training tasks characterized by high and stable workloads and the need to avoid interference, exclusive resource configuration and gang scheduling mechanisms can be activated upon identification to ensure training continuity. The scheduling strategy can accurately match task requirements, avoiding resource waste and stability risks caused by a one-size-fits-all approach, and ensuring stable and efficient operation of both types of tasks.

[0034] In this embodiment, the specific identification method for the task type corresponding to the new pod is not limited. To enhance understanding, four identification methods are introduced here. You can select one or more of them for configuration as needed, or configure other identification methods.

[0035] Method 1: Extract the workload type field from the tags of the new pod. If the workload type field is a preset online development identifier, it is determined to be an online development task; if the workload type field is a preset offline training identifier, it is determined to be an offline training task.

[0036] Method 2: Extract the name of the new pod. If the name starts with a preset online development name prefix, it is determined to be an online development task. If the name starts with a preset offline training name prefix, it is determined to be an offline training task.

[0037] Method 3: Determine the resource request method for the new pod. If the resource request method only sets resource limits and does not set resource requests, it is determined to be an online development task; if the resource request method sets both resource requests and resource limits, it is determined to be an offline training task.

[0038] Method 4: Calculate the GPU utilization of the new pod. If the GPU utilization fluctuates intermittently and the total time in the low utilization state is higher than the preset time threshold, it is determined to be an online development task; if the GPU utilization is consistently higher than the high threshold and the fluctuation is less than the low threshold, it is determined to be an offline training task.

[0039] This embodiment uses the above four methods as examples for description. The settings for other methods can be described in the same way as described in this embodiment, and will not be repeated here.

[0040] Step 103: Develop a GPU scheduling strategy that matches the task type.

[0041] Based on the task type identified in step 102, and considering the inherent load characteristics, resource requirements, and operational scenario demands of online development tasks and offline training tasks, we design specific GPU resource allocation rules, deployment logic, and resource configuration schemes to form a GPU scheduling strategy that is precisely adapted to the task type, thus solving the problem of load and strategy mismatch.

[0042] The specific terms of the scheduling strategies set for the two types of tasks are not limited in this embodiment. They need to be set according to the actual resource status of the cluster, the specific needs of the tasks, and the business priorities. For example, for online development tasks, terms such as prioritizing allocation to nodes that are already running similar tasks and have spare GPUs, allowing GPU resource over-allocation, and setting only resource limits without setting resource requests can be set. For offline training tasks, terms such as prioritizing allocation to nodes where GPUs meet preset idle conditions or are only running similar tasks can be set, and both resource requests and resource limits can be set to achieve exclusive use, distributed tasks can use Gang scheduling to ensure simultaneous scheduling of multiple pods, and multi-GPU tasks can use topology-aware card selection to optimize communication performance can be set. This embodiment only uses the above strategy terms as examples, but it is not limited to them. The settings can be set according to the actual application scenario, and will not be elaborated further here.

[0043] Step 104: Determine whether the available resources meet the requirements of the GPU scheduling policy based on the GPU resource status database; if they do, trigger step 105; if they do not, trigger step 106. Step 105: Execute the scheduling and allocation operation corresponding to the GPU scheduling policy.

[0044] Step 106: Seize online development task resources that meet the preset conditions, triggering step 104.

[0045] Steps 104 to 106 will be described together here. After formulating the dedicated scheduling strategy in step 103, the AI ​​training platform uses the GPU resource status database as data support to first determine whether the available GPU resources in the cluster can meet the core requirements of the scheduling strategy (such as node deployment priority, resource configuration requirements, exclusive / shared mode, etc.). If the resources meet the requirements, the scheduling allocation operation corresponding to the strategy is executed directly. If the resources are insufficient, idle capacity is released by preempting online development task resources that meet the preset conditions. Then, the process returns to the resource judgment stage until the scheduling strategy requirements are met and the scheduling is completed, ensuring that the task can be implemented according to the established strategy, while avoiding task blocking caused by resource supply and demand imbalance.

[0046] Specifically, real-time available resource data for each GPU node (including the number of idle GPUs, occupied resources, node deployment adaptability, etc.) is extracted from the GPU resource status database and precisely compared with the scheduling policy requirements defined in step 103. For example, online development tasks need to determine whether there are nodes that are already running similar tasks and have spare GPUs, while offline training tasks need to determine whether there are nodes that meet the preset idle conditions or only run similar tasks. At the same time, it is verified whether the resource configuration can match the requirements of exclusive / shared modes, and then a judgment result of whether it meets or does not meet is obtained. If the judgment result is met, the rules of the corresponding scheduling policy are followed to complete the implementation actions such as node screening, GPU binding, and resource parameter configuration, ensuring that every clause of the scheduling policy is executed accurately. If the judgment result is not met, the resources of online development tasks (not offline training task resources) are preempted only for resources that meet the preset idle and non-core occupied conditions. After releasing the available GPUs, scheduling is not performed directly, but step 104 is triggered to re-judge the resources, avoiding invalid operations caused by the resources still not meeting the requirements after preemption, forming a retry mechanism of judgment-preemption-re-judgment.

[0047] This method avoids the problem of allocating resources solely based on static requests, as is found in existing methods, when scheduling GPUs. Instead, it uses the real-time available resources displayed in the GPU resource status database as the objective basis and a dedicated scheduling strategy as the demand standard. This ensures that resource judgment is based on real-time status and precisely aligned with the dedicated scheduling strategy. Furthermore, this method overcomes the pain point of blindly waiting or evicting resources when resources are insufficient in traditional methods. By preempting online development task resources through targeted operations, it leverages the characteristics of online development task resources—large fluctuations and the ability to be reused during off-peak periods—to prioritize the preemption of idle resources from online development tasks. This not only solves the resource gap of offline training tasks but also aligns with the load characteristics of online development tasks, avoiding the waste of resources that are occupied but not used. It releases capacity without affecting core training tasks, rather than indiscriminately evicting low-priority resources, thereby improving the overall resource turnover rate of the cluster.

[0048] Based on the above introduction, the intelligent GPU resource allocation method for AI training platforms provided in this embodiment periodically collects multi-dimensional operating status data of GPU nodes and classifies and stores it to generate a resource status database, ensuring that the cluster GPU resource status is real-time and verifiable; it listens to pod creation events, captures key pod creation events, and identifies two core tasks: online development and offline training. This overcomes the limitation of existing technologies that cannot perceive differences in load characteristics, providing a core basis for formulating differentiated scheduling strategies and avoiding resource conflicts caused by treating different loads indiscriminately; then, it addresses the issues of large resource fluctuations, low utilization and long-term occupation of online development tasks, and high load and stable occupation of offline training tasks, requiring... By avoiding the inherent characteristics of interference, a customized scheduling logic is developed to solve the problem of single and rigid scheduling strategies in existing technologies. This achieves precise adaptation of scheduling rules, ensuring that resource allocation is highly compatible with task requirements. Finally, based on the GPU resource status library, the available resources are identified. If the available resources can meet the requirements of the current GPU scheduling strategy, the adapted scheduling is executed directly to ensure rapid task deployment. If resources are insufficient, idle resources from eligible online development tasks are preempted. This not only solves the resource gap in offline training tasks but also maximizes the reuse of idle resources occupied by online development tasks, avoiding resource waste and task blocking, and balancing resource utilization and task operation stability.

[0049] This method achieves an efficient balance between GPU resource utilization and task operation stability on the AI ​​training platform through a closed-loop mechanism that includes multi-dimensional GPU resource data collection, accurate task type identification, differentiated scheduling strategy formulation, and targeted preemption of idle online development resources.

[0050] Example 2: The specific details of the GPU scheduling strategies formulated for online development tasks and offline training tasks in the above embodiments are not limited. In order to maintain the flexibility and adaptability of the solution and to cope with the differences in cluster size, resource distribution, business priority and load characteristics of different AI push training platforms, this embodiment proposes a specific strategy setting method, which can provide an executable and implementable scheduling basis that accurately matches the load characteristics of the two types of tasks.

[0051] Specifically, if the task type is an online development task, a centralized deployment strategy is generated. The centralized deployment strategy includes: prioritizing allocation to nodes that have already run similar tasks and have spare GPUs, allowing GPU resource over-allocation, and configuring resources by setting resource limits without setting resource requests.

[0052] The "prioritizing allocation to nodes already running similar tasks with remaining GPUs" strategy employs a bin-packing centralized deployment logic. This involves scanning the cluster for nodes already hosting online development tasks and with available GPUs, prioritizing new online development tasks to these nodes. This avoids tasks being scattered across multiple nodes, allows for reserving complete nodes for offline training tasks, and reduces the probability of mixed deployment of the two types of tasks. The "allowing GPU resource over-allocation" strategy leverages the low actual utilization rate of online development tasks (often below 20%), which are mostly idle or under low load. This allows the GPU resources of a single node to be used for more than its physical number of online development tasks. Task allocation, such as assigning a node with 8 GPUs to 10 online development tasks, relies on staggered task usage to achieve resource reuse, avoiding resource contention and improving resource turnover efficiency. Configuring resources by setting only resource limits without setting resource requests means limiting the upper limit of GPU resources that a task can use through the Limit parameter, without setting the Request parameter (i.e., not reserving dedicated resources). This ensures that when a task is not using the GPU, the resources can be dynamically occupied by other tasks, breaking the resource monopoly restriction and allowing idle GPUs to be reused by other tasks. This solves the problem of wasted resources that are occupied but not used, and is suitable for non-continuously occupied use cases.

[0053] In the online development scenario of the AI ​​training platform, users' GPU usage needs vary significantly. Some users require exclusive use of the entire GPU to ensure computational efficiency when performing large-scale data preprocessing and complex model debugging, while others only need to perform lightweight code verification and small-scale experiments, requiring only shared or partial GPU computing power. Having defined the core logic of centralized nodes, allowing over-allocation, and non-exclusive configuration in the centralized deployment strategy for online development tasks, this embodiment further proposes a scheduling allocation operation execution method to better adapt to diverse user needs. Specifically, step 104, executing the scheduling allocation operation corresponding to the GPU scheduling strategy, can be broken down into the following sub-steps: Step 41: Select target nodes according to the centralized deployment strategy and receive the GPU usage mode specified by the user.

[0054] First, based on the core rules of the strategy (prioritizing allocation to nodes that have already run similar online development tasks and have spare GPUs), the GPU nodes of the AI ​​training platform are screened to ultimately determine the target nodes that meet the deployment requirements. At the same time, the platform provides a user interaction portal to receive GPU usage modes actively specified by users according to their own task needs (such as task complexity and computing power requirements), including dedicated full-card mode, shared mode, and virtual GPU mode.

[0055] It should be noted that this embodiment only uses the above three usage modes as examples. Other modes can be further configured according to different usage needs, and this embodiment does not limit this.

[0056] Step 42: If the GPU usage mode is full-card exclusive mode, set the GPU resource configuration parameters of the target node to the number of full cards specified by the user.

[0057] When the user specifies the GPU usage mode as the dedicated GPU mode, based on the user's explicit requirement for the number of GPUs, such as 1 or 2 GPUs, the GPU resource configuration parameters corresponding to the selected target nodes will be directly set to the specific number of GPUs specified by the user. This ensures that the user's online development tasks can exclusively use the requested complete GPU resources, avoiding computing power interference caused by sharing with other tasks. This is to adapt to online development scenarios that require high computing power, such as large-scale data preprocessing and complex model debugging.

[0058] Step 43: If the GPU usage mode is shared mode, set the GPU resource configuration parameters of the target node to the preset sharing identifier, and share the GPU through GPU virtualization technology.

[0059] When the user specifies the GPU usage mode as shared mode, the GPU resource configuration parameters of the target node are first set to the platform's preset sharing identifier (used to trigger the GPU sharing mechanism). Then, a resource sharing channel is built through GPU virtualization technology (such as NVIDIA MPS), allowing the user's online development tasks to share the same GPU resources of the target node with other online development tasks that have also selected the shared mode. This is to adapt to scenarios such as lightweight code verification and small-scale experiments that do not require exclusive use of the entire GPU's computing power.

[0060] Step 44: If the GPU usage mode is virtual GPU mode, set the GPU resource configuration parameters of the target node to the virtual GPU share split according to the preset ratio, and allocate and use the virtual GPU according to GPU slicing technology.

[0061] When the user specifies the GPU usage mode as virtual GPU mode, the GPU resources of the target node are divided into corresponding virtual GPU shares according to the platform's preset computing power split ratio (such as splitting at a ratio of 50% or 25%). Then, through GPU slicing technology (such as NVIDIA vGPU), the split virtual GPU shares are allocated to the user's online development tasks, realizing the fine-grained splitting and use of GPU resources. This adapts to lightweight online development tasks that only require a portion of the GPU computing power to complete, avoiding the waste of computing power caused by the occupation of the entire card's resources.

[0062] The scheduling and allocation method achieves tiered reuse of GPU resources through differentiated resource configuration in three modes: dedicated full-card mode ensures stable computing power for core tasks, shared mode enables multiple tasks to be used on the same card during off-peak hours using GPU virtualization technology, and virtual GPU mode completes fine-grained splitting of computing power through slicing technology. These three modes cover all scenarios from high computing power to lightweight, maximizing the potential of GPU resources in centralized deployment nodes. This can solve the contradiction of insufficient computing power for large tasks and wasted computing power for small tasks, balancing users' personalized computing power needs with the platform's overall resource optimization goals, and significantly improving resource utilization efficiency and user experience in online development scenarios.

[0063] If the task type is an offline training task, a dedicated exclusive deployment strategy is generated. The dedicated exclusive deployment strategy includes: prioritizing allocation to GPU nodes that meet preset idle conditions or only run similar tasks, configuring resources and setting resource requests and resource limits.

[0064] The preset idle condition refers to GPU utilization being below a preset threshold (e.g., 5%) for a sustained period of time (e.g., 30 minutes), while excluding false idle conditions. Prioritize nodes of this type or those running only other offline training tasks to avoid resource contention caused by co-location with online development tasks. This reduces the risk of training interruptions or performance fluctuations, especially if online development tasks experience sudden high loads and preempt resources. Configuring resources involves setting resource requests and limits: reserving dedicated GPU resources for the task using the Request parameter and limiting resource usage using the Limit parameter. This ensures the task has exclusive access to the requested GPU resources during runtime, providing a stable supply of GPU resources to meet the high-intensity usage demands during training, where GPU utilization is typically above 70%, thus preventing training failures or extended durations due to insufficient resources.

[0065] In the offline training scenario of the AI ​​training platform, offline training tasks have two core forms: single-machine and distributed. Distributed offline training tasks (such as multi-machine and multi-card training) rely on multiple related pods (including parameter server pods, multiple worker node pods, etc.) to work together. All related pods are required to complete scheduling, startup and obtain dedicated resources at the same time. Otherwise, some pods may occupy resources and wait idly, while some pods may fail to start due to insufficient resources, which may eventually lead to training task blocking, resource deadlock or data synchronization failure. Existing technologies lack a dedicated safeguard mechanism for the multi-pod collaborative scheduling characteristics of distributed training. To address the technical pain point of asynchronous scheduling of associated pods in distributed scenarios and to avoid limiting the effectiveness of dedicated deployment strategies, task type judgment and adaptation configuration can be further refined within the execution framework of dedicated deployment strategies. Specifically, it can first determine whether the offline training task is a single-machine or distributed type. If it is a distributed offline training task, batch scheduling annotation configuration can be added to the dedicated deployment strategy. By specifying the batch scheduling enable flag (explicitly triggering the batch scheduling mechanism) and the minimum available number threshold of associated pods (defining the total number of pods that need to be scheduled synchronously), a dual safeguard of dedicated resources and synchronous scheduling is formed.

[0066] For example, a user submitted a distributed offline training task based on the PyTorch framework, consisting of 3 machines and 6 GPUs, for training a large-scale image classification model. This task requires collaboration between one parameter server (PS) pod and three worker pods. The PS pod is responsible for parameter synchronization and distribution, requiring one GPU resource. Each worker pod is responsible for parallel data computation, requiring two GPU resources. All associated pods must start simultaneously and acquire stable resources; otherwise, issues such as parameter synchronization timeouts and training process blocking will occur. For this distributed task, the following batch scheduling annotation configuration is added to the dedicated deployment strategy: `scheduling.volcano.sh / gang:'true'` is set as the batch scheduling enable flag, explicitly triggering the Volcano scheduler's collaborative scheduling mechanism; `scheduling.volcano.sh / min-available:'4'` is set as the minimum available number threshold for associated pods, corresponding to the total number of the 1 PS pod and 3 Worker pods. This configuration, combined with the core rules of the dedicated exclusive strategy, requires the cluster to simultaneously allocate GPU resources that meet the exclusive requirements to the four associated pods (a total of 1 + 3 × 2 = 7 GPUs). All pods must be scheduled to nodes where the GPUs meet the preset idle conditions to avoid co-location with online development tasks. Only when all the above conditions are met will the scheduler uniformly execute the scheduling and allocation operation to ensure that all associated pods start synchronously and acquire dedicated exclusive resources synchronously. This not only ensures the parameter synchronization and consistency of distributed training, but also avoids the scheduling chaos of some pods occupying resources idly and some pods having insufficient resources.

[0067] This batch scheduling annotation configuration directly addresses the core requirement of scheduling all associated pods simultaneously, completely solving the problems of resource deadlock and training interruption caused by asynchronous pod scheduling in existing distributed training technologies. All associated pods start synchronously and acquire exclusive resources synchronously, avoiding the waste of computing power caused by some pods occupying resources idly. At the same time, it ensures the continuity of data synchronization and model training, and reduces the performance loss caused by scheduling delays.

[0068] In offline training scenarios of AI training platforms, offline training tasks not only differ in form (single-machine vs. distributed) but also in resource requirements (single-GPU vs. multi-GPU). The performance of multi-GPU offline training tasks (such as multi-GPU parallel training and distributed multi-GPU collaborative training) is highly dependent on the physical topology between GPUs. If the topological distance between GPU combinations is large (e.g., across PCIe Switches, without NVLink connections), it will significantly increase data communication latency between GPUs, severely reducing training efficiency. In contrast, single-GPU offline training tasks only need to meet basic resource exclusivity requirements to ensure operational stability. To further differentiate the resource allocation details between single-GPU and multi-GPU tasks, a mechanism for adapting GPU topology associations for multi-GPU tasks is proposed, based on the dedicated exclusivity deployment strategy for offline training tasks. This embodiment further proposes an execution method for scheduling and allocation operations. Specifically, for the dedicated exclusivity deployment strategy, step 104 executes the scheduling and allocation operation corresponding to the GPU scheduling strategy, which can be broken down into the following sub-steps: Step 45: Determine whether the number of GPUs used in the dedicated deployment strategy is a single card or multiple cards; if it is a single card, trigger step 46; if it is multiple cards, trigger step 47.

[0069] Step 46: Select target nodes according to the dedicated deployment strategy, and bind the GPUs in the target nodes that meet the resource requirements to the new pod.

[0070] Step 47: Read the GPU topology association data in the GPU resource status library, enumerate the GPU combinations that meet the task's GPU quantity requirements, score each GPU combination according to the topology connection type, and select the GPU combination with the highest score to bind to the new pod.

[0071] Specifically, when executing the scheduling and allocation operation of the dedicated deployment strategy, the number of GPUs required for the offline training task is first extracted from the preset parameters in the strategy to determine whether the task belongs to a single-GPU training scenario that only requires a single GPU or a multi-GPU training scenario that requires two or more GPUs. When the task is determined to be a single-GPU offline training task, the node selection rules of the dedicated deployment strategy are strictly followed (prioritizing nodes whose GPUs meet preset idle conditions or only run similar offline training tasks) to determine the target nodes that meet the requirements; then, from the target nodes, an unoccupied single GPU that meets the task's resource requirements is selected, and a one-to-one resource binding is performed between it and the new pod to be scheduled, ensuring that the pod can exclusively use all the resources of this GPU, thus meeting the basic requirement of resource exclusivity for single-GPU offline training tasks.

[0072] When the task is determined to be a multi-GPU offline training task, the topological association data between GPUs within the target node (such as NVLink connection relationships, PCIe topology, whether they belong to the same PCIeSwitch, etc.) is retrieved from the GPU resource status database generated in step 101. Then, based on the number of GPUs required by the task, all GPU combinations within the target node that meet the requirement are listed (for example, if the task requires 4 GPUs, all combinations of 4 GPUs in the node are enumerated). Next, scoring rules are set according to the quality of the topology connection type (such as GPU combinations with high-speed NVLink connections score higher than combinations connected only through PCIe, and GPU combinations belonging to the same PCIeSwitch score higher than combinations across switches), and each GPU combination is quantitatively scored. Finally, the optimal GPU combination with the highest score is selected and bound to a new pod to ensure that the multi-GPU training task can exclusively use GPU resources while relying on the optimal topology to reduce data communication latency between GPUs and maximize the performance of multi-GPU parallel training.

[0073] The implementation method of this strategy extends the dedicated deployment strategy from node-level exclusive protection to GPU combinatorial-level topology optimization, specifically addressing the pain point of topology adaptation for multi-GPU offline training tasks, while being compatible with the basic requirements of single-GPU tasks. This allows the dedicated strategy to not only guarantee resource exclusivity but also maximize the collaborative performance of multi-GPU training.

[0074] It should be noted that this embodiment only uses the above strategy content as an example for introduction, but is not limited to it. The scheduling and allocation operations under other strategy content can refer to the introduction of this embodiment, and will not be repeated here.

[0075] Based on the above introduction, this embodiment implements differentiated core scheduling strategies for online development and offline training tasks, further refines the multi-mode GPU configuration for online development tasks, the distributed batch scheduling guarantee for offline training tasks, and the multi-card topology-aware card selection logic, and achieves precise adaptation between task requirements and resource configuration. This maximizes GPU resource utilization and comprehensively ensures the operational stability and multi-card training performance of various tasks, efficiently adapting to the diverse needs of AI push training scenarios.

[0076] Example 3: In the resource scheduling scenario of AI training platforms, when offline training tasks are submitted but cluster resources are insufficient, it is necessary to preempt online development task resources that meet preset conditions. Although idle resources can be released in a targeted manner, the common preemption methods in existing technologies are mostly direct expulsion, lacking consideration for user usage status and interactive mechanisms. Online development task users may be in the critical stage of code debugging and experimental verification. Sudden resource preemption can lead to loss of work progress and impaired experience. Moreover, the preemption process lacks transparent recording, making it difficult to trace subsequent problems and affecting the reliability of the platform. To avoid this problem and make the preemption mechanism meet resource scheduling requirements while taking into account user experience and process standardization, this embodiment proposes a resource preemption method. This method accurately selects target pods that meet preset conditions from the GPU resource status database, sends preemption reminders and countdown notifications to the users, and reserves response time. Depending on whether the user actively releases resources, it performs recycling or forced termination operations according to the scenario, and stores complete preemption information throughout the process. This method can minimize the interference with the work of online development users while ensuring the timely supply of offline training task resources, and achieves standardization of the preemption process, humanization of interaction, and traceability of operation.

[0077] Specifically, step 106 involves seizing online development task resources that meet preset conditions, which can be performed according to the following steps: Step 61: Filter target pods that meet preset conditions from the GPU resource status database.

[0078] When performing resource preemption operations for online development tasks, the GPU resource status database is used as data support. Based on preset conditions (such as the pod corresponding to the online development task, GPU utilization being below the threshold for a preset duration, non-core business identifiers, etc.), all pod-related data stored in the database are filtered to accurately locate the target pod that meets the preemption requirements. This clarifies the specific target for subsequent targeted preemption operations, ensuring that the preemption action is only aimed at compliant and releaseable resources, and avoiding the accidental preemption of offline training tasks or online development task resources that are being used frequently.

[0079] The specific conditions set in this embodiment are not limited. In order to balance the flexibility of the solution with the accuracy of the selection, adapt to the differences in cluster size, resource distribution, business priority and user usage scenarios of different AI training platforms, and avoid problems such as false preemption, invalid preemption or user experience damage caused by the rigidity of the selection rules, a target pod selection mechanism is provided here. Through the progressive logic of task type-oriented selection, resource status quantitative verification, false idleness accurate elimination, and preemption cost quantitative sorting, the target pod that is truly idle, has the least impact on users and can effectively release resources can be accurately identified from the massive number of online development task pods, ensuring the rationality, effectiveness and humanization of the preemption operation.

[0080] Specifically, step 61 can be performed as follows: Step 611: Select the pods corresponding to online development tasks from the GPU resource status database as candidate pods.

[0081] Using the GPU resource status database as the data source, and based on the core filtering dimension of task type, pods associated with online development tasks are selected from all pod data stored in the database and designated as candidate pods.

[0082] Step 612: Combining GPU utilization data and idle time records, extract pods from the candidate pods whose GPU utilization is lower than the preset utilization threshold and whose idle time exceeds the preset time threshold, and use them as pre-selected pods.

[0083] Based on the candidate pods identified in step 611, two quantitative indicators, GPU utilization and idle time, are introduced for further screening. Specifically, real-time GPU utilization data and historical idle time records for each candidate pod are retrieved from the GPU resource status database. Pods that simultaneously meet two conditions are selected as pre-selected pods: first, the GPU utilization is consistently lower than the platform's preset utilization threshold (e.g., 20%, indicating that resources are not being fully utilized); second, the idle time exceeds the preset duration threshold (e.g., 30 minutes, indicating that resources are in a low-load idle state for a long period).

[0084] This step uses quantitative indicators to initially identify pods with high resource idleness, avoiding including frequently used online development pods in the preemption scope and improving the rationality of the selection.

[0085] Step 613: Exclude pseudo-idle pods from the pre-selected pods that are in the initial startup phase, process hibernation, or waiting for input / output, and use them as the initial selection pods.

[0086] The pre-selected pods obtained in step 612 are subjected to a true idle verification to exclude falsely idle pods, ultimately resulting in the initial selected pods. False idle refers to a pod that appears to have low load (meeting utilization and duration threshold conditions) but is actually in a temporary state within a normal usage process. This includes three scenarios: the initial low-load phase during pod startup, the process sleep waiting to be woken up, and the temporary pause phase waiting for input / output (I / O). The specific filtering conditions for these three scenarios are not limited in this embodiment and can be configured according to the actual application scenario.

[0087] This step can eliminate pods that are not truly free of resources, preventing accidental preemption that could interrupt users' normal development work and further improving the accuracy of the screening.

[0088] Step 614: Calculate the preemption cost of each initially selected pod using the preset formula, and select the pod with the lowest preemption cost as the target pod.

[0089] For the initial pods obtained in step 613, the preset preemption cost calculation formula is called. The core logic of this formula is to positively weight the pod runtime (the longer the runtime, the higher the preemption cost, which means that the user may have a higher dependence on the pod) and negatively weight the GPU idle time corresponding to the pod (the longer the idle time, the lower the preemption cost, which means that the user has a lower dependence on the pod). The quantified cost score of each initial pod is obtained through weighted calculation.

[0090] Sort all the initial pods by cost score and select the pod with the lowest cost as the final target pod.

[0091] This step involves quantitatively assessing the balance between resource release demand and user experience impact, ensuring that the preemptive operation has minimal impact on users, and achieving the optimal balance between resource allocation and user rights.

[0092] It should be noted that this embodiment only uses the above screening steps as an example for introduction, but is not limited to this. The implementation methods of screening conditions set for other purposes can refer to the introduction of this embodiment, and will not be repeated here.

[0093] Step 62: Send a preemption reminder and resource release countdown notification to the user who owns the target pod, and wait for the preset response time.

[0094] Once the target pod is identified, the platform sends a clear preemption reminder (informing the user of the reason for the resource being preempted) and a resource release countdown notification (indicating the remaining response time) to the user of the target pod via preset notification channels (such as platform in-app messages, emails, and SMS). The pod then enters a waiting state until the preset response time expires. The core of this step is to provide users with buffer time, allowing them to save their current work progress and manually release resources, avoiding work loss due to sudden preemption, thus balancing resource scheduling needs with user experience.

[0095] Step 63: If the resources corresponding to the target pod are released within the preset response time, then resource reclamation is initiated.

[0096] Within the preset response time set in step 62, if it is detected that the user to which the target pod belongs has actively released the corresponding GPU resources (such as manually closing the pod and releasing GPU usage), the resource reclamation process is immediately started to update the released GPU resources from the occupied state to the available state and synchronize them to the GPU resource status database, so as to prepare for the subsequent reassessment of whether the resources meet the requirements of the offline training task scheduling strategy.

[0097] Step 64: If the resources corresponding to the target pod are not released within the preset response time, send a deletion signal to the target pod and set a termination time. After the termination time is reached, forcibly terminate the process of the target pod and store the corresponding preemption information.

[0098] If no resources corresponding to the target pod are actively released within the preset response time set in step 62, a deletion signal is sent to the target pod according to the process, and a reasonable graceful termination time is set (ensuring that the pod has time to complete the basic state saving); when the termination time is reached, the running process of the target pod is forcibly terminated, the GPU resources it occupies are released and the GPU resource state database is updated synchronously; at the same time, all relevant information of this preemption is fully stored, including but not limited to: the timestamp of the preemption, the name and identifier of the preempted pod, the user information, the specific reason for the preemption (such as meeting the resource requirements of high-priority offline training tasks), and the relevant information of the newly scheduled pod, to ensure that the preemption operation is traceable and auditable.

[0099] This embodiment accurately identifies truly idle online development task pods through multi-dimensional, layer-by-layer screening. It combines the quantification of preemption costs to select targets with the least impact on users, along with a flexible interactive mechanism of reminder notifications and response buffers, and a standardized recycling / forced termination process. At the same time, it fully records preemption information, ensuring the timely release of resources required for offline training tasks while minimizing interference with users. This achieves precision, user-friendliness, and standardization in preemption operations, balancing resource scheduling efficiency and user rights, and improving the reliability and traceability of the platform's scheduling mechanism.

[0100] Example 4: In the above embodiments, resource release for online development tasks relies on passive preemption triggering. That is, idle resources of online development tasks are only released through preemption when offline training tasks are submitted and resources are insufficient. However, in practical applications, online development tasks often encounter scenarios such as users forgetting to release resources, tasks remaining idle for extended periods after being paused, and experiments not being cleaned up after completion. This results in some GPU resources remaining idle for extended periods without being actively reclaimed, leading to hidden resource waste. This not only reduces the overall utilization rate of the cluster's GPUs but also indirectly increases the frequency of resource preemption for offline training tasks, potentially causing more user experience conflicts. To address this pain point of passive management, this embodiment proposes an active resource reclamation method based on idle state monitoring. This method can further mine the value of idle resources by leveraging the collected multi-dimensional GPU operating status data.

[0101] Specifically, in addition to the steps mentioned above, the following steps may be performed: Step 107: Determine the idle GPUs in the online development task based on the multi-dimensional running status data of the GPU.

[0102] Based on the multi-dimensional runtime status data (such as GPU real-time utilization, process occupancy status, matching degree between resource requests and actual usage, task execution logs, etc.) collected and stored in the GPU resource status database, the system accurately identifies GPUs in the online development tasks that are not used by actual computing processes and are in an idle state through data analysis and status verification. It then filters out the implicitly idle resources from the GPU resources of online development tasks, providing clear targets for subsequent proactive reclamation.

[0103] Step 108: Determine whether the idle time of the idle GPU reaches the idle threshold standard corresponding to the online development task. If it does, trigger step 109.

[0104] After identifying an idle GPU in step 107, the historical idle time record of that GPU is retrieved and compared with a preset idle threshold standard adapted to the usage characteristics of online development tasks to determine whether its cumulative idle time has reached or exceeded the threshold. The idle threshold standard corresponding to online development tasks is set in conjunction with the specific characteristics of online development scenarios (such as distinguishing between temporary idleness during short-term debugging intervals and permanent idleness due to long-term forgetting). The specific value is not limited in this embodiment to avoid interfering with normal user development due to the reclamation of short-term idle resources. If the idle time reaches the threshold, it indicates that the GPU has been in a long-term idle state and has reclamation value, thereby triggering subsequent gradient reclamation steps.

[0105] Step 109: Perform gradient-based resource reclamation on idle GPUs.

[0106] When an idle GPU reaches its reclamation threshold, instead of a direct, forceful release, a gradient-based resource reclamation operation is performed. This gradient refers to a phased reclamation process. For example, in the first phase, an idle reminder and a resource retention countdown notification are sent to the user; in the second phase, if the user does not respond, the GPU's resource allocation permissions are restricted, and over-allocation reuse is stopped; in the third phase, after the countdown ends, GPU resources are released and the resource state database is updated. Specific gradient settings are not limited in this embodiment.

[0107] By gradually tightening resource permissions and reserving multiple user response windows, the resource recycling needs and user experience are balanced. While actively revitalizing idle resources, the interference with users' normal development work is minimized, ensuring the humanization and controllability of the recycling operation.

[0108] If the idle time of the idle GPU does not reach the idle threshold standard corresponding to the online development task, this embodiment does not limit the handling method in this case. It can maintain the current resource occupation status of the idle GPU and continue to reserve resources for the corresponding online development task to avoid affecting the user's normal development due to short-term idle misoperation; or, based on the principle of resource optimization, allow the GPU to participate in the over-allocation reuse of the online development task (that is, it can still be shared by other lightweight online development tasks) to improve resource turnover efficiency without interfering with the original task; or simply record the idle status and cumulative duration of the GPU, continuously monitor its idle time changes until the threshold standard is reached before triggering the gradient recycling process; or send a gentle idle reminder to the user, allowing the user to choose whether to release resources or extend the retention time, etc.

[0109] This embodiment actively revitalizes long-term idle resources by accurately identifying idle GPUs in online development tasks and triggering gradient recycling, transforming hidden waste into usable computing power. This can effectively improve the overall utilization rate of cluster GPUs and alleviate the contradiction between resource supply and demand.

[0110] Example 5: To enhance understanding, this embodiment describes a complete implementation process, including component deployment, data collection, scheduling decisions, resource adaptation, continuous optimization, and high availability assurance, referencing... Figure 2 The details are as follows: First, complete the deployment of core components, deploying the resource exporter as a daemon set on each GPU node to ensure a single monitoring instance per node; at the same time, deploy a resource manager developed based on a container orchestration custom resource and operator framework to build the core architecture of monitoring and decision-making.

[0111] Secondly, real-time monitoring and data reporting of GPU resources are initiated. The resource exporter collects GPU hardware layer data, topology data, container GPU usage and load type data at fixed time intervals. Static data is stored through node tags and dynamic data is stored through custom resources. Simultaneously, fine-grained idle detection is achieved, idle time is accumulated and false idle scenarios are eliminated, forming a complete GPU resource status data system.

[0112] Next, the resource manager triggers scheduling decisions by listening to events such as container creation, changes in resource metrics, and changes in container status. It first identifies the task type as online development or offline training by label, name prefix, and resource request method, and then executes differentiated scheduling strategies: online development tasks are deployed centrally, allow over-allocation, and have non-exclusive resource configuration; offline training tasks are given priority for idle nodes and have strict exclusive resource configuration; if it is distributed training, batch scheduling is enabled to ensure synchronous scheduling of associated containers.

[0113] If resources are insufficient when an offline training task is submitted, an intelligent preemption process is executed: first, online development task containers that have been idle for a long time are selected and the preemption cost is calculated based on runtime and idle time, and the target container with the lowest cost is selected; preemption is executed through gentle notification or forced eviction, reserving a buffer time for user response, and training tasks are scheduled after resources are released and the preemption history is fully recorded.

[0114] For multi-GPU training tasks, topology-aware scheduling is implemented: read node topology information, enumerate GPU combinations that meet the quantity requirements and score them according to connection type, select the optimal combination to bind to the container, and integrate a scheduler scoring plugin to achieve topology optimization.

[0115] After task scheduling is completed, the runtime continuous optimization phase begins: the resource exporter sends notifications to users according to the idle time gradient and performs automatic recycling when there is heavy idle time; the resource manager monitors the cluster load and achieves dynamic load balancing by prioritizing the scheduling of new tasks; when a GPU hardware anomaly is detected, container migration is triggered to ensure task continuity.

[0116] For online development containers, the configuration resource is in a non-exclusive mode, supporting three usage modes: dedicated GPU, shared GPU, and virtual GPU. Users can specify the GPU model to adapt to diverse development needs.

[0117] Finally, high availability is achieved through resource manager multi-replica election, monitoring data access performance is optimized through local caching, and scheduling efficiency is improved through incremental scheduling and concurrent decision-making to ensure stable and efficient system operation.

[0118] Example 6: This embodiment relates to a GPU resource intelligent allocation device for an AI training platform. A schematic diagram of the GPU resource intelligent allocation device for this embodiment can be seen as follows: Figure 3 As shown, it includes: a data acquisition unit 201, a pod creation and identification unit 202, a scheduling strategy formulation unit 203, a real-time resource judgment unit 204, a scheduling and allocation unit 205, and a resource preemption unit 206.

[0119] Among them, the data acquisition unit 201 is used to periodically collect multi-dimensional GPU operating status data of each GPU node in the AI ​​training platform, classify and store it, and generate a GPU resource status library. The pod creation recognition unit 202 is used to listen for pod creation events. When a pod creation event is detected, it identifies the task type corresponding to the new pod. The task types include online development tasks and offline training tasks. The scheduling policy formulation unit 203 is used to formulate a GPU scheduling policy that matches the task type; The real-time resource judgment unit 204 is used to determine whether the available resources meet the requirements of the GPU scheduling policy based on the GPU resource status library; if they meet the requirements, the scheduling allocation unit 205 is triggered; if they do not meet the requirements, the resource preemption unit 206 is triggered. The scheduling and allocation unit 205 is used to execute the scheduling and allocation operations corresponding to the GPU scheduling policy. The resource preemption unit 206 is used to preempt online development task resources that meet preset conditions and then trigger the real-time resource judgment unit 204.

[0120] The intelligent GPU resource allocation device for the AI ​​training platform provided in this embodiment constructs a multi-dimensional GPU resource status database through an independent data acquisition unit, providing comprehensive and real-time data support for scheduling decisions and ensuring the accuracy of resource judgment. A pod creation and identification unit accurately identifies two core task types: online development and offline training. Combined with differentiated strategy design by the scheduling strategy formulation unit, this achieves precise matching between task requirements and resource allocation. A real-time resource judgment unit assesses resources; when resources are sufficient, the scheduling allocation unit directly executes adaptive scheduling; when resources are insufficient, the resource preemption unit preempts idle online development resources, ensuring resource supply for high-priority tasks while avoiding resource waste. Each functional unit has clearly defined responsibilities and works collaboratively and efficiently, forming a fully automated process from data acquisition to scheduling implementation. This reduces manual intervention, significantly improves the utilization efficiency and scheduling reliability of GPU resources in the AI ​​training platform, and adapts to the dynamic resource needs of diverse tasks.

[0121] Additionally, it should be noted that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units.

[0122] Furthermore, in order to highlight the innovative aspects of this application, no units that are not closely related to solving the technical problems proposed in this application are introduced in this embodiment, but this does not mean that there are no other units in this embodiment.

[0123] Example 7: Another embodiment of this application relates to an electronic device, such as... Figure 4 As shown, it includes: at least one processor 301; and a memory 302 communicatively connected to at least one processor 301; wherein the memory 302 stores instructions executable by at least one processor 301, the instructions being executed by at least one processor 301 to enable at least one processor 301 to perform the steps of the GPU resource intelligent allocation method for AI training platform in the above embodiments.

[0124] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0125] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0126] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A method for intelligent allocation of GPU resources for an AI training platform, characterized in that, include: Regularly collect multi-dimensional GPU operating status data from each GPU node in the AI ​​training platform, classify and store the data, and generate a GPU resource status database. Listen for pod creation events. When a pod creation event is detected, identify the task type corresponding to the new pod. The task types include: online development tasks and offline training tasks. Develop a GPU scheduling strategy that matches the task type; The system determines whether the available resources meet the requirements of the GPU scheduling policy based on the GPU resource status database. If they do, the system performs the scheduling and allocation operation corresponding to the GPU scheduling policy. If they do not meet the requirements, the system preempts online development task resources that meet the preset conditions and then performs the step of determining whether the available resources meet the requirements of the GPU scheduling policy based on the GPU resource status database.

2. The intelligent GPU resource allocation method for an AI training platform according to claim 1, characterized in that, The process of formulating a GPU scheduling strategy that matches the task type includes: If the task type is an online development task, a centralized deployment strategy is generated; the centralized deployment strategy includes: prioritizing allocation to nodes that have already run similar tasks and have remaining GPUs, allowing GPU resource over-allocation, and configuring resources by setting resource limits without setting resource requests; If the task type is an offline training task, a dedicated exclusive deployment strategy is generated; the dedicated exclusive deployment strategy includes: prioritizing allocation to GPU nodes that meet preset idle conditions or only run similar tasks, configuring resources while setting resource requests and resource limits.

3. The intelligent GPU resource allocation method for an AI training platform according to claim 2, characterized in that, If the GPU scheduling policy is the centralized deployment policy, then the execution of the scheduling and allocation operation corresponding to the GPU scheduling policy includes: The target nodes are selected according to the centralized deployment strategy, and the GPU usage mode specified by the user is received. If the GPU usage mode is a dedicated full-card mode, the GPU resource configuration parameters of the target node are set to the number of full cards specified by the user; If the GPU usage mode is shared mode, the GPU resource configuration parameters of the target node are set to a preset sharing identifier, and GPU sharing is performed through GPU virtualization technology; If the GPU usage mode is virtual GPU mode, the GPU resource configuration parameters of the target node are set to a virtual GPU share divided according to a preset ratio, and the virtual GPU is allocated and used according to GPU slicing technology.

4. The intelligent GPU resource allocation method for an AI training platform according to claim 2, characterized in that, Also includes: Determine whether the offline training task is a single-machine offline training task or a distributed offline training task; If it belongs to the distributed offline training task, add batch scheduling annotation configuration to the dedicated exclusive deployment strategy; The batch scheduling annotation configuration includes: specifying the batch scheduling enable flag and the minimum available number threshold of associated pods required for the task, so as to ensure that all associated pods are scheduled at the same time.

5. The intelligent GPU resource allocation method for an AI training platform according to claim 2, characterized in that, If the GPU scheduling policy is the dedicated exclusive deployment policy, then the execution of the scheduling and allocation operation corresponding to the GPU scheduling policy includes: Determine whether the GPU usage in the exclusive deployment strategy is a single card or multiple cards; If it is a single card, the target node is selected according to the exclusive deployment strategy, and the GPU in the target node that meets the resource requirements is bound to the new pod; If there are multiple GPUs, read the GPU topology association data in the GPU resource status database, enumerate the GPU combinations that meet the task's GPU quantity requirements, score each GPU combination according to the topology connection type, and select the GPU combination with the highest score to bind to the new pod.

6. The intelligent GPU resource allocation method for an AI training platform according to claim 1, characterized in that, The process of seizing online development task resources that meet preset conditions includes: Filter target pods that meet the preset conditions from the GPU resource status database; Send a preemption reminder and a resource release countdown notification to the user to which the target pod belongs, and wait for a preset response time; If the resources corresponding to the target pod are released within the preset response time, then resource reclamation will be initiated. If the resources corresponding to the target pod are not released within the preset response time, a deletion signal is sent to the target pod and a termination time is set. After the termination time is reached, the process of the target pod is forcibly terminated, and the corresponding preemption information is stored. The preemption information includes: timestamp, information of the preempted pod, user information, reason for preemption, and information of the newly scheduled pod.

7. The intelligent GPU resource allocation method for an AI training platform according to claim 6, characterized in that, The step of filtering target pods that meet the preset conditions from the GPU resource status database includes: Pods corresponding to online development tasks are selected from the GPU resource status database as candidate pods; Combining GPU utilization data and idle time records, pods with GPU utilization below a preset utilization threshold and idle time exceeding a preset time threshold are extracted from the candidate pods and used as pre-selected pods. Exclude pseudo-idle pods from the pre-selected pods that are in the initial startup phase, process hibernation, or waiting for input / output, and use them as the initial selection pods; The preemption cost of each initially selected pod is calculated by calling a preset formula, and the pod with the lowest preemption cost is selected as the target pod. The preset formula is obtained by performing a positive correlation weighted calculation on the pod runtime and a negative correlation weighted calculation on the GPU idle time corresponding to the pod.

8. The intelligent GPU resource allocation method for an AI training platform according to claim 1, characterized in that, Also includes: Idle GPUs in online development tasks are determined based on the multi-dimensional GPU operating status data. Determine whether the idle time of the idle GPU reaches the idle threshold standard corresponding to the online development task; If this is achieved, the idle GPU will be reclaimed in a gradient manner.

9. A GPU resource intelligent allocation device for an AI training platform, characterized in that, include: The data acquisition unit is used to periodically collect multi-dimensional running status data of each GPU node in the AI ​​training platform, classify and store it, and generate a GPU resource status library. A pod creation identification unit is used to listen for pod creation events. When a pod creation event is detected, the unit identifies the task type corresponding to the new pod. The task types include online development tasks and offline training tasks. The scheduling strategy formulation unit is used to formulate a GPU scheduling strategy that matches the task type. The real-time resource judgment unit is used to determine whether the available resources meet the requirements of the GPU scheduling policy based on the GPU resource status database; if they meet the requirements, the scheduling allocation unit is triggered; if they do not meet the requirements, the resource preemption unit is triggered. The scheduling and allocation unit is used to execute the scheduling and allocation operation corresponding to the GPU scheduling policy; The resource preemption unit is used to preempt online development task resources that meet preset conditions, and then trigger the real-time resource judgment unit.

10. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the GPU resource intelligent allocation method for an AI training platform as described in any one of claims 1 to 8.

Images