K8S multi-machine multi-card distributed training method, device and equipment

Abstract

The embodiment of the invention relates to the technical field of container group management, in particular to a K8S multi-machine multi-card distributed training method and device and electronic equipment. The method comprises the following steps: continuously monitoring a creation event of a custom training task in an API server of the K8S; when the creation event is monitored, extracting user-defined training configuration information; matching a corresponding distributed training framework according to a framework type in the training configuration information; based on the training configuration information and the distributed training architecture, generating operation configuration information of distributed training adaptive to the distributed architecture; the training configuration information and the operation configuration information are submitted to a customized scheduler, and resource scheduling and creation of container groups and service resources corresponding to the working nodes are carried out; and after each container group is started, controlling each working node to cooperatively execute a training task based on the operation configuration information. According to the method, the configuration threshold of the distributed training task can be reduced, and the configuration complexity of a user on bottom K8S resources is reduced.

Description

Technical Field

[0001] This application relates to the field of container group management technology, and in particular to a K8S multi-machine multi-card distributed training method, device and electronic device. Background Technology

[0002] With the continuous growth of deep learning model scale and data volume, multi-machine, multi-GPU distributed training has become a core technical path to improve training efficiency. It involves forming a cluster of servers equipped with multiple GPUs, leveraging parallel computing and parameter synchronization mechanisms to collaboratively complete model training, breaking through the computing power limitations of a single GPU / machine. Kubernetes (K8S), as an open-source container orchestration platform, has gradually become the mainstream deployment carrier for distributed training tasks due to its flexible resource management capabilities. Currently, the mainstream distributed training implementation solutions mainly fall into three categories: traditional cluster management methods (relying on tools such as Slurm, requiring manual configuration of node networks and roles), cloud platform managed solutions (bound to specific cloud services, lacking cross-environment migration flexibility), and native K8S solutions (relying on open-source projects such as Kubeflow, providing basic distributed training capabilities).

[0003] However, existing technologies have significant drawbacks: First, resource configuration is static, requiring users to manually preset CPU, memory, GPU, and other resource requirements. Over-configuration can lead to resource waste, while under-configuration can cause task failure, and there is a lack of dynamic adjustment mechanisms at runtime. Second, GPU resource utilization is low. Due to the lack of task queue management, priority preemption, and automatic replacement mechanisms, a large number of GPUs are running inefficiently or idle, and users cannot know the task queue position or estimated waiting time. Third, scheduling strategies are not well adapted to distributed training scenarios. The default K8S scheduling strategy is prone to resource fragmentation and does not consider the communication characteristics of the framework, resulting in high latency between nodes. Parallel tasks in the same training pipeline may also experience IO contention due to centralized scheduling. Fourth, task configuration has a high barrier to entry. Users need to write complex K8S native resource configuration files and manually configure the environment variables required for distributed training, which is cumbersome and prone to errors.

[0004] Therefore, how to simplify distributed training tasks while accurately adapting them to training scenarios is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide at least one K8S multi-machine multi-card distributed training method, device and electronic device, which can reduce the configuration threshold of distributed training tasks and reduce the configuration complexity of underlying K8S resources for users.

[0006] To address the aforementioned technical problems, at least one embodiment of this application provides a K8S multi-machine, multi-GPU distributed training method, comprising: Continuously monitor the creation event of the custom training task in the Kubernetes API server; When the creation event is detected, the fields in the custom training task are parsed to extract the user-defined training configuration information; Match the corresponding distributed training architecture based on the framework type in the training configuration information; Based on the training configuration information and the distributed training architecture, generate distributed training runtime configuration information adapted to the distributed architecture; The training configuration information and the runtime configuration information are submitted to the customized scheduler for resource scheduling and the creation of container groups and service resources corresponding to the worker nodes. After each container group is started, it controls each of the working nodes to collaboratively execute training tasks based on the running configuration information.

[0007] In one embodiment, after controlling each of the working nodes to collaboratively execute the training task based on the runtime configuration information, the method further includes: Collect resource usage data of the container group; Based on the resource usage data, the ratio of the actual resource usage to the configured resource request amount for each resource is calculated as the resource request rate. Determine whether the resource request rate has reached the expansion threshold; if so, perform resource expansion. Determine whether the resource request rate is lower than the reduction threshold. If it is, perform resource reduction.

[0008] In one embodiment, the method for performing the resource expansion or resource reduction includes: Save the current training checkpoint file; The new resource configuration parameters are calculated based on the resource application rate and the preset adjustment rules. Based on the new resource configuration parameters, create a new container group instance; Once the new container group instance reaches the ready state, the training checkpoint file will be migrated to the new container group instance. Delete the original running container group instance and update the configuration of the corresponding service resources so that the service resources point to the new container group instance.

[0009] In one embodiment, before submitting the training configuration information and the runtime configuration information to the customized scheduler for resource scheduling and the creation of container groups and service resources corresponding to worker nodes, the method further includes: Read the GPU queuing enable configuration from the training configuration information; If the enabled configuration is enabled, the priority calculation algorithm is invoked to calculate the priority result in combination with the task information; the task information includes: user-set priority, task waiting time, user quota usage and task type. The training task is added to the GPU task queue corresponding to the priority based on the priority result; Monitor the idle status of the GPU resources and the waiting status of training tasks in each queue; When there are available GPU resources, the first schedulable task is selected from the corresponding queue in descending order of priority and scheduled to be executed on the available GPU resources.

[0010] In one embodiment, after monitoring the idle state of GPU resources and the waiting state of training tasks in each queue, the method further includes: If there are tasks waiting in the high-priority queue and no free GPU resources, check the tasks being executed in the low-priority queue and select the target task from low to high priority. After saving the checkpoint file of the target task, stop the target task and release the resources, and schedule high-priority waiting tasks to be executed on the released resources; The target task is then prioritized and added back to the original queue.

[0011] In one embodiment, the K8S multi-machine multi-card distributed training method further includes: Based on the remaining time of the currently running task, the total duration of the preceding tasks in the queue, and the GPU resource release rate, the estimated waiting time of each task is calculated and generated, and added to the GPU task queue.

[0012] In one embodiment, the process of controlling each of the working nodes to collaboratively execute the training task based on the runtime configuration information further includes: Detect the GPU usage status of each container group and calculate the GPU utilization. When the GPU utilization rate is lower than a preset threshold, GPU sharing configuration is executed so that multiple training tasks can share the same GPU resources.

[0013] In one embodiment, the training configuration information and the runtime configuration information are submitted to a customized scheduler for resource scheduling and the creation of container groups and service resources corresponding to worker nodes, including: The customized scheduler calculates the CPU request rate, memory request rate, and GPU request rate of each node; The weighted overall request rate of the computing nodes is calculated based on the CPU request rate, the memory request rate, and the GPU request rate. The scheduling score for each node is calculated based on the overall request rate, and the container groups corresponding to each working node are preferentially scheduled to the nodes with higher scheduling scores; the lower the overall request rate, the higher the scheduling score.

[0014] At least one embodiment of this application also provides a K8S multi-machine multi-card distributed training device, comprising: The task monitoring module is used to continuously monitor the creation event of custom training tasks in the K8S API server; when the creation event is detected, the configuration extraction module is triggered. The configuration extraction module is used to parse the fields in the custom training task and extract user-defined training configuration information. The architecture matching module is used to match the corresponding distributed training architecture based on the framework type in the training configuration information. The runtime configuration generation module is used to generate runtime configuration information for distributed training that is adapted to the distributed architecture, based on the training configuration information and the distributed training architecture. The resource scheduling module is used to submit the training configuration information and the running configuration information to the customized scheduler to perform resource scheduling and create container groups and service resources corresponding to the worker nodes; The task coordination execution module is used to control each of the working nodes to coordinately execute training tasks based on the running configuration information after each of the container groups is started.

[0015] At least one embodiment of this application also provides an electronic device, including: at least one processor; and a memory communicatively connected to said at least one processor; The memory stores instructions that can be executed by the at least one processor, which are then executed by the at least one processor to enable the at least one processor to execute the K8S multi-machine multi-card distributed training method.

[0016] The K8S multi-machine, multi-GPU distributed training method provided in this application automatically listens for task creation events, parses user configuration requirements, accurately matches the distributed architecture according to the training framework type, and automatically generates runtime configuration information, ensuring deep compatibility between the configuration and the architecture and framework communication characteristics. The training and runtime configuration information are submitted to a customized scheduler, which performs resource scheduling and automated creation of container groups and service resources corresponding to worker nodes according to the user-defined training tasks. Users do not need to manually write underlying K8S resource configurations such as Pods and Services, completely shielding the user from underlying technical details. After each container group starts, it controls each worker node to automatically read the runtime configuration information and automatically triggers collaborative execution of training tasks through preset collaborative control logic. This method, through full-process automation design and declarative management, lowers the threshold for distributed training configuration. Users only need to initiate custom training tasks. Relying on framework-adaptive architecture matching and standardized collaborative execution, it achieves platformization, efficiency, and stability for multi-machine, multi-GPU distributed training. Attached Figure Description

[0017] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0018] Figure 1 This is a flowchart illustrating a K8S multi-machine, multi-card distributed training method according to an embodiment of this application; Figure 2 This is a schematic diagram of the deployment and status of a K8S worker node layer provided in one embodiment of this application; Figure 3 This is a schematic diagram of the overall functional architecture of a GPU task queue manager provided in one embodiment of this application; Figure 4 This is an embodiment of the overall flowchart of distributed training task submission and scheduling provided in this application; Figure 5 This is a schematic diagram of a K8S multi-machine multi-card distributed training device provided in one embodiment of this application; Figure 6 This is a schematic diagram of the module division of another K8S multi-machine multi-card distributed training device provided in one embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0020] The following is a detailed description of the implementation details of the K8S multi-machine multi-card distributed training method in this embodiment. The following content is only for the convenience of understanding the implementation details and is not necessary for implementing this solution.

[0021] Example 1: The specific process of the K8S multi-machine multi-card distributed training method in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 101: Continuously listen for the creation event of the custom training task in the K8S API server.

[0022] In the Kubernetes API server, a custom training task refers to a distributed training task configuration declared based on a Custom Resource Definition (CRD) within the Kubernetes (K8S) container orchestration platform's API server (kube-api server). This is a high-level business resource object designed to simplify user operations and adapt to multi-machine, multi-GPU distributed training scenarios. By extending the K8S API, complex distributed training requirements (such as framework selection, number of nodes, resource quotas, etc.) are abstracted into standardized declarative configurations, rather than users directly writing native K8S Pod and Service configurations.

[0023] It continuously monitors the creation of new custom training tasks (CRDs) in the K8SAPI server. When a task creation event that meets the preset specifications is detected, it automatically triggers the entire process of subsequent task parsing, configuration generation, resource scheduling, etc., without the need for manual initiation of the deployment process.

[0024] It should be noted that the above example uses a custom training task configured via CRD, but it is not limited to this. For example, K8SYAML configuration (Pod, Service, Config Map) can also be used.

[0025] Step 102: When the creation event is detected, parse the fields in the custom training task and extract the user-defined training configuration information.

[0026] After capturing the creation event of a custom training task (i.e., training task CRD) in the K8SAPI server, the configuration fields of the task are parsed in a structured manner. From the high-level semantic configuration declared by the user, the core requirement information supporting the subsequent training process is accurately extracted. The user's business requirements are transformed into standardized data that the system can recognize and execute, providing basic input for subsequent operations such as distributed architecture matching and runtime configuration generation.

[0027] The core configuration fields of this custom training task (key information that users need to declare) include, but are not limited to: framework type (such as PyTorch, TensorFlow, Horovod, etc.), number of worker nodes, resource requirements per node (CPU, memory, number of GPU cards), container image (including training code and dependent environment), training start command and hyperparameters (learning rate, batch size and other training hyperparameters), and optional advanced configurations (GPU queuing enabled, resource adaptive adjustment switch, affinity mode, etc.).

[0028] Step 103: Match the corresponding distributed training architecture based on the framework type in the training configuration information.

[0029] Traditional solutions require users to manually select the distributed architecture and configure component roles based on the framework type. This can easily lead to communication failures or low training efficiency due to architecture-framework mismatch. This step, based on the framework type (such as PyTorch, TensorFlow, Horovod, etc.) extracted in step 102, automatically matches a distributed training architecture that is compatible with the communication characteristics and computational logic of the framework using preset mapping rules. For example, when the framework type is PyTorch, it matches the Master-Worker architecture; when the framework type is TensorFlow, it matches the PS-Worker (parameter server-worker node) architecture; and when the framework type is Horovod or MPI, it matches the Launcher-Worker architecture. In this embodiment, there are no restrictions on the framework type and the corresponding mapping rules. The configuration can be made according to the actual application scenario. Here, only the above three framework types are used as examples. Other mapping rules can be referred to the description in this embodiment and will not be repeated here.

[0030] This method eliminates the need for user intervention in architecture selection and adaptation. The system automatically matches the architecture based on the framework type, resulting in a high degree of automation and lowering the technical barrier to distributed training. Furthermore, pre-defined mapping rules based on framework communication characteristics ensure a high degree of compatibility between the architecture and the framework, avoiding issues such as high communication latency and training interruptions caused by architecture incompatibility. The mapping rule library supports flexible expansion, allowing for the addition of new framework-architecture mappings to adapt to more distributed training frameworks such as MXNet, thus meeting the diverse technical selection needs of different users.

[0031] Step 104: Based on the training configuration information and the distributed training architecture, generate the running configuration information for distributed training adapted to the distributed architecture.

[0032] Based on the training configuration information extracted in step 102 (such as the number of worker nodes, resource requirements, framework type, etc.) and the distributed training architecture matched in step 103 (such as Master-Worker, PS-Worker, etc.), standardized configuration data that is fully adapted to the architecture and can directly support the operation of training tasks is automatically generated, transforming the abstract task requirements and architecture rules into specific configurations that can be recognized and executed by the container group.

[0033] The runtime configuration information focuses on the core configuration items adapted to the distributed architecture, mainly including two parts: first, the exclusive environment variables of each worker node (such as RANK, WORLD_SIZE, MASTER_ADDR, MASTER_PORT, etc., used to define the node role, communication address and cluster size); second, the affinity strategy adapted to the characteristics of the framework (such as the same-node affinity strategy for the PyTorch architecture and the different-node anti-affinity strategy for the TensorFlow architecture).

[0034] Among them, the multi-level affinity scheduling strategy is a scheduling optimization scheme designed to optimize worker node distribution, reduce communication latency, and avoid resource contention, taking into account the differences in communication characteristics of different distributed training frameworks. The communication mode and frequency of different distributed training frameworks determine their optimal worker distribution: For GPU training tasks in PyTorch (using the Ring AllReduce communication mode) and Horovod (using the MPI mode), since workers need to conduct high-frequency ring communication for each batch, the optimal distribution is to have the same node to minimize communication latency; For TensorFlow (using the PS-Worker communication mode) or CPU training tasks, since they rely on load-balanced communication between workers and parameter servers (PS), the optimal distribution is to have different nodes to achieve balanced utilization of computing resources. Based on these differences, this strategy automatically generates adaptive affinity configurations through algorithms: for GPU training tasks like PyTorch and Horovod, Pod affinity rules are enabled to prioritize scheduling workers for the same training task to the same node; for TensorFlow or CPU training tasks, Pod anti-affinity rules are enabled to prevent workers for the same task from being concentrated on a single node. Simultaneously, to address the IO contention issue of multiple parallel tasks within the same training pipeline, this strategy also adds a pipeline-level anti-affinity mechanism. This mechanism uses label matching (based on pipeline-id and task-id) to distribute different tasks within the same pipeline to different nodes, preventing disk IO or network IO congestion. Its core innovation lies in intelligently generating strategies by combining the framework's communication characteristics. This ensures optimal distribution when resources are sufficient and supports degraded scheduling when resources are insufficient. Furthermore, through a multi-level affinity combination at the worker, pipeline, and node levels, it achieves synergistic optimization of communication efficiency and resource utilization.

[0035] To deepen understanding, a specific form of runtime configuration information is introduced here, taking the training configuration information extracted in step 102 as an example, where the framework type is PyTorch, the number of worker nodes is 4, and step 103 matches a Master-Worker distributed architecture: One set of environment variable generation results is as follows: Worker-0 has RANK=0, WORLD_SIZE=4, MASTER_ADDR=worker-0.llama2-svc.default.svc.cluster.local, and MASTER_PORT=23456; Worker-1 to Worker-3 have RANK values ​​of 1-3 respectively, and WORLD_SIZE, MASTER_ADDR, and MASTER_PORT are consistent with Worker-0. Only the RANK field is assigned values ​​incrementally according to the node number, which clarifies the cluster role and communication address of each node. One affinity strategy generates the following result: Based on the Ring All Reduce communication feature of the PyTorch framework, a Pod affinity configuration is generated. The label Selector is used to match the label of training-job:llama2. With kubernetes.io / hostname as the topology key, all worker nodes are scheduled to the same physical node first, thereby reducing the communication latency between nodes.

[0036] Additionally, it should be noted that since the types of information contained in the runtime configuration information differ under different architectures, the above does not limit the specific architecture type of the distributed architecture, nor does it limit the specific information type of the runtime configuration information corresponding to different distributed architectures. All can refer to the above introduction, and will not be repeated here.

[0037] Step 105: Submit the training configuration information and runtime configuration information to the customized scheduler to perform resource scheduling and create the container groups and service resources corresponding to the worker nodes.

[0038] The training configuration information extracted in step 102 (such as the number of worker nodes and resource requirements) and the runtime configuration information generated in step 104 (such as environment variables and affinity policies) are integrated into a standardized scheduling request and submitted to a customized scheduler adapted to the distributed training scenario. The scheduler then completes the precise selection and resource allocation of cluster nodes and triggers the automatic creation of container groups (Pods) and service resources corresponding to the worker nodes, transforming the abstract task configuration into actual computing resources that can run in the cluster.

[0039] The customized scheduler refers to a dedicated scheduling component developed based on the native Kubernetes scheduler (kube-scheduler) to adapt to the resource scheduling needs of multi-machine, multi-GPU distributed training scenarios, achieving the dual goals of efficient resource allocation and stable task deployment. Since different distributed training architectures have different resource types, the specific working process of the customized scheduler is not limited here. To deepen understanding, the resource scheduling and creation process of the container groups and service resources corresponding to the worker nodes are introduced. Taking the PyTorch framework + Master-Worker architecture (4 worker nodes, 1 GPU per node) as an example, the resource scheduling process is as follows: The customized scheduler receives the training configuration (number of Workers = 4, single Worker GPU = 1) and the running configuration (same-node affinity policy, environment variable template). Calculate the resource request rate of each node in the cluster: Assume that the cluster has 3 nodes (Node-1~Node-3), each node has a GPU capacity of 4, and the current GPU request rates are Node-1 (0%), Node-2 (25%), and Node-3 (50%). Node-1 is scored according to the weighted overall request rate (GPU weight 0.4, CPU / memory weight 0.3 each). Based on the Ring All Reduce communication feature of the PyTorch framework, and by matching the same node affinity strategy, it was confirmed that Node-1 can accommodate the GPU requirement of 4 Workers (4 idle GPUs). Check if the resources of Node-1 meet the total requirements of all 4 Workers (4 GPUs + 32 CPUs + 128 Gi memory). Once confirmed, schedule them in batches to avoid scheduling failures for some Workers.

[0040] The result of creating a container group (Pod) and service resources is as follows: Create 4 Worker Pods (Worker-0 to Worker-3) on Node-1, and inject the exclusive environment variables generated in the runtime configuration into each Pod, for example: Worker-0: RANK=0, WORLD_SIZE=4, MASTER_ADDR=worker-0.llama2-svc.default.svc.cluster.local, MASTER_PORT=23456; Worker-1 to Worker-3: RANK is 1 to 3 respectively, and WORLD_SIZE and MASTER_ADDR / PORT are consistent with Worker-0 to ensure clear communication identification between nodes; Each Pod has resource limits consistent with the training configuration (CPU=8, memory=32Gi, GPU=1) and is bound to the training-job:llama2 tag for affinity policy verification.

[0041] Create a Headless Service named llama2-svc, associate it with 4 Worker Pods, resolve the network address of the Pods through the Service, ensure the communication stability under the Master-Worker architecture, and avoid connection interruption caused by changes in Pod IP.

[0042] Step 106: After each container group starts up, it controls each working node to collaboratively execute training tasks based on the running configuration information.

[0043] Once all the container groups (Pods) corresponding to all worker nodes in the K8S cluster have started successfully, the preset distributed collaborative control logic drives each worker node to strictly follow the running configuration information generated in step 104 to complete node identity verification, distributed communication link construction, parallel splitting of training tasks and parameter synchronization, etc., and finally realize multi-machine and multi-card collaborative training, transforming the previous configuration parsing, architecture matching and resource scheduling results into the actual execution process of model training.

[0044] Based on the above introduction, the K8S multi-machine, multi-GPU distributed training method provided in this embodiment automatically listens for task creation events, parses user configuration requirements, accurately matches the distributed architecture according to the training framework type, and automatically generates runtime configuration information, ensuring deep compatibility between configuration and architecture, and framework communication characteristics. The training configuration information and runtime configuration information are submitted to a customized scheduler, which performs resource scheduling and automated creation of container groups and service resources corresponding to worker nodes according to the user-defined training tasks. Users do not need to manually write underlying K8S resource configurations such as Pods and Services, completely shielding the user from underlying technical details. After each container group starts, it controls each worker node to automatically read the runtime configuration information and automatically triggers collaborative execution of training tasks through preset collaborative control logic. This method, through full-process automation design and declarative management, lowers the threshold for distributed training configuration. Users only need to initiate custom training tasks. Relying on framework-adaptive architecture matching and standardized collaborative execution, it achieves platformization, efficiency, and stability for multi-machine, multi-GPU distributed training.

[0045] Example 2: In existing Kubernetes multi-machine, multi-GPU distributed training scenarios, traditional solutions rely on users manually configuring CPU, memory, and other resource requirements when submitting tasks. However, users often find it difficult to accurately predict the actual resource consumption during training, leading to the prominent pain point of static resource configuration: over-configuration will waste cluster resources and reduce overall utilization; under-configuration is prone to interruption of training tasks due to memory overflow (OOM) or insufficient CPU, and lacks a runtime dynamic adjustment mechanism, requiring manual intervention to modify the configuration and resubmit, which seriously affects training efficiency and continuity.

[0046] Considering that during the execution of training tasks, factors such as data loading speed and model computational complexity can cause dynamic changes in resource usage, a simple initial configuration cannot adapt to real-time requirements. Therefore, this embodiment proposes to add a dynamic resource adjustment step after collaborative training execution. By sensing the resource usage status in real time, intelligent adaptive optimization of resource configuration can be achieved, which avoids the inherent defects of static configuration and does not require manual intervention, ensuring the continuous and stable progress of training tasks.

[0047] Specifically, after step 106 controls each working node to collaboratively execute the training task based on the runtime configuration information, the following steps can be further performed: Step 107: Collect resource usage data for the container group.

[0048] It captures in real time the actual resource consumption data of the container group (Pod) corresponding to each worker node during the training process, such as the actual CPU usage, actual memory usage, actual GPU usage, and video memory usage.

[0049] By acquiring real-time data on changes in resource consumption during training, the system can accurately grasp the dynamics of resource usage, providing data support for subsequent intelligent adjustments to adapt to dynamic needs.

[0050] Step 108: Based on the resource usage data, calculate the ratio of the actual resource usage to the allocated resource request amount for each resource, which is used as the resource request rate.

[0051] For various resources such as CPU and memory, obtain the resource request amount configured for each, and execute the calculation logic of actual resource usage ÷ configured resource request amount. For example, CPU request rate = actual CPU usage / CPU configured request amount, and memory request rate = actual memory usage / memory configured request amount.

[0052] By calculating the ratio, the resource usage status is transformed into a quantitative indicator in the range of 0-1, providing a clear basis for decision-making regarding expansion / reduction.

[0053] Step 109: Determine whether the resource request rate has reached the expansion threshold. If it has, execute resource expansion.

[0054] The resource request rate is compared with the preset expansion threshold. When the resource request rate reaches or exceeds the expansion threshold, the resource expansion operation is automatically triggered to ensure that the training task has sufficient resources to support it and avoid training interruption due to insufficient resources.

[0055] Step 110: Determine if the resource request rate is lower than the reduction threshold. If it is, execute resource reduction.

[0056] The resource request rate is compared with a preset reduction threshold. When the resource request rate is lower than the reduction threshold, the resource reduction operation is automatically triggered to release redundant resources for other tasks, thereby improving the overall resource utilization of the cluster.

[0057] In this embodiment, the specific numerical configuration of the expansion threshold and reduction threshold is not limited. For example, the expansion threshold can be set to 90% and the reduction threshold can be set to 50%. When the resource utilization rate is too low (<50%), the configuration is reduced to release resources. When the resource utilization rate is too high (>90%), the configuration is expanded to avoid OOM. When the resource utilization rate is moderate (50%-90%), no adjustment is made to avoid frequent adjustments.

[0058] This method relies on a real-time resource data acquisition mechanism to capture the actual consumption status of resources such as CPU and memory during container group training. Then, by calculating the resource request rate between actual usage and configuration requests, it quantifies resource usage status into a unified evaluation index, providing a clear basis for scaling up or down. Based on preset thresholds, it automatically triggers scaling up or down operations, expanding in a timely manner when the resource request rate reaches the threshold and releasing redundant resources when the resource request rate falls below the threshold, achieving dynamic adaptation of resource configuration to actual needs. The entire adjustment process requires no manual intervention, automating the process while helping distributed training platforms achieve more efficient and stable resource management.

[0059] This embodiment does not limit the specific operation methods for resource expansion and reduction. For example, it may use methods such as directly terminating the original container group and rebuilding a new container group or forcibly modifying resource parameters in the original container group and restarting it. However, the former will lead to the loss of unsaved training progress and the interruption of distributed communication links, while the latter is prone to container group restart failure due to resource adjustments, and the service will be unresponsive during the restart, directly causing the training task to be interrupted. In order to solve the core pain points of the existing execution method, such as difficulty in ensuring training continuity, easy damage to data integrity, and lack of awareness of service switching, and to ensure that training progress is not lost, communication links are not interrupted, and service access is not abnormal during dynamic resource adjustment, this embodiment further proposes an implementation method for resource expansion and reduction. The execution method for resource expansion or reduction can be performed according to the following steps: Step 111: Save the current training checkpoint file.

[0060] Before resource adjustments, retain the current progress and core data of the training task, including key information such as model parameters, optimizer status, and number of training iterations, and form checkpoint files (such as .pth and .ckpt formats) and store them in shared storage or a specified directory to avoid loss of training progress when switching container groups later.

[0061] Step 112: Calculate the new resource configuration parameters based on the resource application rate and preset adjustment rules.

[0062] Based on the resource request rate (actual usage / configuration request) calculated in step 108, new resource allocation is quantified by combining preset rules.

[0063] The adjustment rules for expansion and reduction are not limited here. For example, when expanding, the calculation can be based on the actual usage multiplied by 1.3 (leaving a 30% redundancy to avoid resource shortages again), and when reducing, the calculation can be based on the actual usage multiplied by 1.2 (retaining a 20% safety margin to ensure training stability). This ensures that the new configuration adapts to actual needs while balancing stability and resource utilization. Of course, other adaptive adjustment rules can also be set according to different node types or functional requirements, which will not be elaborated here.

[0064] Step 113: Create a new container group instance based on the new resource configuration parameters.

[0065] Based on the new resource configuration parameters (such as 10 CPU cores and 42Gi memory after expansion, or 6 CPU cores and 28Gi memory after reduction), create a new container group (Pod) in the K8S cluster, and inherit the training environment, image information and running configuration (such as environment variables and communication rules) of the original container group to ensure that the new container group has the basic conditions for continuing training.

[0066] Step 114: Once the new container group instance is ready, migrate the training checkpoint files to the new container group instance.

[0067] After the new container group has finished starting (its status is Ready, meaning the training environment and dependent components are ready), move the checkpoint file saved in step 111 from shared storage or the original container group directory to the specified path of the new container group to ensure that the new container group can directly load the historical training progress without restarting the training.

[0068] Step 115: Delete the original running container group instance and update the configuration of the corresponding service resources so that the service resources point to the new container group instance.

[0069] First, delete the original container group to release redundant resources, then update the configuration of the associated service resources (Service), and point the network access address of the service to the new container group to ensure that the communication link between nodes in the distributed training is not interrupted, and the new container group can be seamlessly connected to the cluster for collaborative training.

[0070] It should be noted that the above methods can be applied to both resource expansion and reduction, and the core execution logic of the two is completely consistent.

[0071] This method achieves seamless integration of resource adjustment and training execution through a closed-loop process of checkpoint preservation, new instance pre-creation, smooth migration, and old instance cleanup, without requiring manual intervention throughout the entire process. It perfectly supports intelligent decision-making for adaptive resource allocation and can further enhance the stability and efficiency of distributed training tasks.

[0072] Example 3: In existing Kubernetes (K8S) multi-machine, multi-GPU distributed training scenarios, the native K8S scheduling mechanism lacks dedicated queuing management capabilities for GPU resources, resulting in prominent core pain points: when GPU resources are scarce, multiple user-submitted training tasks will remain in a pending state for a long time, and users cannot know the task queue position or estimated waiting time, lacking clear expectations; at the same time, task scheduling only relies on the resource idle state without priority distinction, high-priority production tasks cannot preempt low-priority experimental task resources, and after a completed task releases the GPU, tasks in the queue must wait for the K8S default scheduling loop to fill the gap, resulting in idle and wasted resources.

[0073] To address this issue and avoid resource contention and scheduling chaos, this embodiment proposes to add a GPU task queuing scheduling step before submitting a scheduling request. By constructing a multi-priority task queue mechanism, the orderly allocation, priority control, and intelligent replacement of GPU resources can be achieved. This not only solves the problems of disordered task waiting and low resource utilization in traditional scheduling, but also provides users with transparent queue status awareness, ensuring the priority execution of important tasks.

[0074] Specifically, before submitting the training configuration information and runtime configuration information to the customized scheduler in step 105 to perform resource scheduling and create the container groups and service resources corresponding to the worker nodes, the following steps can be performed first: Step 116: Read the GPU queuing enable configuration from the training configuration information.

[0075] The training configuration information reads the user's preset GPU queuing enable / disable configuration to determine whether to trigger the subsequent queue management process. If it is disabled, it directly enters the normal scheduling process; if it is enabled, it starts the queuing scheduling logic to ensure that the queuing mechanism is enabled as needed.

[0076] Step 117: If the configuration is enabled, call the priority calculation algorithm to calculate the priority result in combination with the task information.

[0077] Once GPU queuing is confirmed, the system invokes a preset priority calculation algorithm to quantify the priority based on key dimensions of the task information. This task information includes, but is not limited to, user-set priority, task wait time, user quota usage, and task type. The priority calculation algorithm may use weighted calculations, such as a 40% weight for user-set priority and a 30% weight for wait time, to provide an objective basis for subsequent queue allocation.

[0078] Step 118: Add the training task to the GPU task queue corresponding to the priority based on the priority result.

[0079] Based on the priority results calculated in step 117, the current training task is categorized into the corresponding priority queue, such as a high-priority queue corresponding to a high-priority result. The specific priority type and number are not limited in the above steps. Since the GPU task queues are configured one-to-one with the priority type, the number of GPU task queues is also not limited; for example, three levels of GPU task queues—high, medium, and low—can be preset.

[0080] To ensure users understand the estimated waiting time of each task in the queue, these times can be added to each GPU task queue. Based on the remaining time of the currently running task, the total duration of preceding tasks in the queue, and the GPU resource release rate, the estimated waiting time for each task is calculated and generated, and then added to the GPU task queue. By integrating key variables affecting waiting time, a quantified estimated waiting time is provided for each queued task, allowing users to clearly understand the task scheduling rhythm. This also provides data support for user decisions (such as adjusting task priorities and choosing submission times) and platform scheduling optimization, further improving the user-friendliness and efficiency of the scheduling mechanism while ensuring orderly scheduling. Of course, estimated waiting times can also be added; this embodiment does not limit this.

[0081] Step 119: Monitor the idle status of GPU resources and the waiting status of training tasks in each queue.

[0082] Continuously monitor the usage and idle status of GPU resources in the cluster (such as the GPU utilization rate of each node and the number of idle GPUs), and track the waiting time and status changes of tasks in each level of queue (such as new tasks, tasks being preempted, tasks being canceled, etc.) to ensure timely capture of resource release and task status change signals, providing real-time data support for subsequent scheduling triggers.

[0083] Step 120: When there are available GPU resources, select the first schedulable task from the corresponding queue in descending order of priority and schedule it to be executed on the available GPU resources.

[0084] When GPU resources are released (e.g., GPUs released after tasks have been completed or resources released by preempted tasks), the system selects the first schedulable task that meets the resource requirements according to the rule of prioritizing high-priority queues and prioritizing tasks that were enqueued first in the same queue. The task is then directly scheduled to be executed on an idle GPU resource without waiting for the K8S default scheduling loop, thus avoiding resource idleness. At the same time, it ensures that high-priority tasks get resources first, improving scheduling efficiency and rationality.

[0085] This method triggers queue management on demand by reading the GPU queuing activation status in the training configuration, adapting to the scheduling needs of different users. It calculates priorities based on user-set priorities, task wait times, user quota usage, and task type, ensuring high-priority tasks have priority while preventing low-priority tasks from being starved due to wait times, thus balancing scheduling fairness and rationality. Through priority queue classification management, users clearly understand task scheduling expectations, monitor GPU resource idle status and queue task dynamics in real time, and automatically fill in scheduling slots according to priority as soon as resources are released, without waiting for the Kubernetes default scheduling loop, significantly reducing resource idle time. This method, through multi-level queues and priority mechanisms, meets the different scheduling needs of production and experimental tasks, and, combined with queue management, ensures orderly task flow, further enhancing the GPU resource management capabilities and scheduling flexibility of the distributed training platform.

[0086] While GPU queuing and scheduling solutions address the issues of unordered task waiting and untimely resource replenishment, under conditions of sustained GPU resource scarcity (e.g., cluster GPUs running at full load for extended periods), high-priority tasks (such as production deployment tasks) may remain in a waiting state due to low-priority tasks (such as experimental testing tasks) occupying resources for extended periods, thus failing to obtain timely computing power support and impairing the timeliness of core business operations. To further improve GPU resource priority management capabilities, resolve the resource scarcity issue for high-priority tasks, and simultaneously protect the legitimate rights of low-priority tasks, this embodiment proposes that after monitoring the idle status of GPU resources and the waiting status of training tasks in each queue in step 119, the following steps can be further performed: Step 121: If there are tasks waiting in the high-priority queue and no free GPU resources, check the tasks being executed in the low-priority queue and select the target task from low to high priority.

[0087] When the system detects that there are tasks to be executed in the high-priority queue and there are no available GPU resources in the cluster, the preemption logic is triggered: the system traverses all running tasks in the low-priority queue, sorts them in order of priority from low to high, and selects the task with the lowest priority as the preemption target.

[0088] This filtering logic minimizes the impact of preemption on other tasks, avoids chain scheduling fluctuations caused by preempting high-priority low-queue tasks, and ensures the rationality of preemption actions.

[0089] Step 122: After saving the checkpoint file of the target task, stop the target task and release the resources, and schedule the high-priority waiting tasks to be executed on the released resources.

[0090] To avoid losing the training results of the target task, the checkpoint saving mechanism of the target task is first triggered to fully preserve the current training progress, including model parameters, optimizer status, iteration count, etc.; then the target task is stopped and the GPU and associated computing resources it occupies are released; finally, the task ranked first in the high-priority waiting queue is immediately scheduled to the newly released GPU resources to start execution, quickly responding to the computing power needs of high-priority tasks and avoiding resource idleness.

[0091] Step 123: Increase the priority of the target task and then add it back to the original queue.

[0092] To prevent low-priority tasks from being repeatedly preempted and unable to progress for extended periods, the system automatically increases the priority of a task after it stops (e.g., by increasing its priority score by 20 points) and then adds it back to its original priority queue, rather than downgrading it or re-queuing it to the end of the queue. This design ensures that preempted tasks receive higher priority scheduling opportunities later, guaranteeing both the priority execution rights of high-priority tasks and the legitimate right of low-priority tasks to proceed, thus avoiding an imbalance in scheduling fairness.

[0093] like Figure 2The diagram illustrates the deployment and state of a Kubernetes worker node layer. A worker node primarily comprises three tasks: Worker Pod deployment tasks for regular distributed training, running GPU training tasks, and queued training tasks. The Worker Pod deployment tasks for regular distributed training clearly define the resource bindings (e.g., specific GPUIDs) and distributed environment variables (e.g., RANK, WORLD_SIZE) of each Worker within the node. This is the core execution carrier supporting multi-machine, multi-GPU distributed training, as described in the previous embodiment, and will not be repeated here. The running GPU training tasks are associated with the specific GPU resources used and the running time, intuitively reflecting the current node's computing power load status. The queued training tasks are marked with task priorities and estimated waiting times, providing a clear representation of the priority task advancement mechanism in this embodiment and reflecting the task queue management logic under GPU resource constraints.

[0094] Figure 3The diagram illustrates the overall functional architecture of the GPU Task Queue Manager, covering queue management, scheduling decisions, preemption, and state control for GPU tasks. It presents the entire process logic of the GPU Task Queue Manager, from task hierarchy and scheduling decisions to preemption and state control. The system divides tasks into high, medium, and low-priority queues, each clearly labeled with the task ID, priority score, and estimated waiting time (e.g., the high-priority queue contains Task-A, priority 80, estimated waiting time 10 minutes), enabling hierarchical and categorized task management and providing a queue foundation for subsequent scheduling decisions. The scheduling decision engine comprises two core computational logics: a priority calculation logic that uses user-set priority, task waiting time, user quota usage, and task type as core dimensions, configuring corresponding weights (e.g., user-set priority weight 40%), and derives a quantified priority score through weighted calculation; and an estimated waiting time algorithm that calculates and generates the estimated waiting time for tasks based on three variables: the remaining time of the currently running task, the total duration of preceding tasks in the queue, and the GPU resource release rate, and synchronizes this information to the user to clarify scheduling expectations. Priority preemption and automatic replacement clearly define the complete preemption and replacement process. The preemption logic is as follows: when there are waiting tasks in the high-priority queue and no idle GPUs, a low-priority task is selected to run. Its checkpoint file is saved, the task is stopped, and resources are released. The high-priority task is then scheduled to that resource. The replacement logic is as follows: after GPU resources are released, the first schedulable task is selected according to the rule of prioritizing high-priority queues and tasks enqueued earlier in the same queue. Simultaneously, the preempted task is promoted in priority and re-added to its original queue. The task state machine displays the state transitions throughout the entire task lifecycle: after submission, the task enters the Queued state; after scheduling is triggered, it transitions to the Scheduling state; after resource binding is complete, it enters the Running state; if preempted, it switches to Preempted; and finally, after completion, it becomes Completed, clearly demonstrating the state change logic of tasks in the queue manager.

[0095] This method addresses the issue of waiting tasks in a high-priority queue with no available GPUs. It preempts low-priority tasks to ensure that core production and urgent tasks quickly acquire computing power, meeting business timeliness requirements. Before preemption, the method forcibly saves the target task's checkpoint file, preserving key data such as model parameters and training iteration progress. Subsequent preempted tasks can seamlessly continue training based on these checkpoints, avoiding waste of previous computational gains. Preempted low-priority tasks are automatically prioritized and re-enqueued, effectively mitigating the starvation problem caused by repeated preemption of the same task and protecting the legitimate right of low-priority tasks to proceed. This method enhances the flexibility of priority scheduling while ensuring the overall stability and fairness of distributed training tasks.

[0096] Example 4: During training, due to factors such as data loading speed, the appropriateness of batch size settings, and fluctuations in model computational complexity, the GPUs in some container groups are often under low load (actual utilization is only 30%-50%), with a large amount of computing power being idle. To fully utilize idle computing power and improve resource reuse, a GPU sharing configuration triggering step is added to the collaborative training execution phase. By detecting GPU usage status in real time, low-utilization scenarios are accurately identified and the sharing mechanism is dynamically activated, ensuring the training progress and stability of the original tasks while maximizing the use of idle computing power and reducing cluster computing costs.

[0097] Specifically, in step 106, when controlling each working node to collaboratively execute training tasks based on the running configuration information, the GPU usage status of each container group can be further detected and the GPU utilization rate can be calculated. When the GPU utilization rate is lower than a preset threshold, GPU sharing configuration is executed so that multiple training tasks can share the same GPU resources.

[0098] This method monitors the GPU utilization of each container group in real time, using a preset threshold (e.g., 50%) as the criterion to ensure that the sharing mechanism is triggered only when the GPU is truly idle. It opens up and shares low-load GPU resources that were previously exclusive to a single task, allowing lightweight training tasks to reuse idle computing power and significantly improving the overall GPU utilization of the cluster. The shared configuration can employ process isolation mechanisms, using resource quota limits (such as memory isolation and compute resource sharding) to avoid computational contention and data interference between multiple tasks, ensuring the training stability of both the original and shared tasks. This method reduces the computational cost per training task by efficiently reusing existing resources, while also adapting to the low-cost execution requirements of lightweight tasks, further enhancing the resource management and cost optimization capabilities of the distributed training platform.

[0099] Example 5: In the scheduling scenario of multi-machine, multi-GPU distributed training in Kubernetes, although the customized scheduler undertakes the core responsibility of resource scheduling, the existing default scheduling strategy of Kubernetes mainly scores nodes based on the amount of resource requests. It tends to distribute container groups to nodes with low resource requests, resulting in resource fragmentation. For example, container groups with multiple worker tasks are split into multiple nodes, and large-scale training tasks that require continuous and complete resources cannot be scheduled due to the lack of available nodes, which seriously affects the efficient reuse of cluster resources.

[0100] To address the resource fragmentation issue caused by traditional scheduling strategies and ensure that scheduling decisions are more adapted to the resource requirements of distributed training scenarios, this embodiment further proposes a scheduling scoring logic based on the overall request rate in a customized scheduler. By quantifying the resource utilization efficiency of nodes (rather than simply the number of requests), the system guides container groups to centrally schedule resources to nodes with low resource request rates, reserving full resources for subsequent large tasks. Simultaneously, it ensures balanced utilization of various resources such as CPU, memory, and GPU, improving scheduling rationality and overall cluster resource utilization.

[0101] Specifically, step 105, which submits the training configuration information and runtime configuration information to the customized scheduler for resource scheduling and the creation of container groups and service resources corresponding to worker nodes, can be performed according to the following steps: Step 51: The customized scheduler calculates the CPU request rate, memory request rate, and GPU request rate for each node.

[0102] The customized scheduler traverses all candidate nodes in the Kubernetes cluster and calculates the request rates for three types of core resources: CPU request rate, memory request rate, and GPU request rate. It focuses on key resources for distributed training and accurately reflects the actual utilization rate of node resources, rather than simply the absolute amount of remaining resources.

[0103] The specific calculation formulas for CPU request rate, memory request rate, and GPU request rate are not limited in this embodiment. One calculation method is as follows: CPU request rate = Total CPU resources already requested by the node ÷ Total CPU capacity of the node; Memory request rate = Total memory resources already requested by the node ÷ Total memory capacity of the node; GPU request rate = Total number of GPUs already requested by the node ÷ Total number of GPUs of the node. Of course, other quantitative calculation methods can also be adopted, all of which can be referred to the description in this embodiment, and will not be elaborated here.

[0104] Step 52: Calculate the weighted overall request rate of the computing nodes based on the CPU request rate, memory request rate, and GPU request rate.

[0105] The scheduler uses a weighted average algorithm to calculate the overall request rate. The weight allocation can fully adapt to the characteristics of distributed training scenarios. For example, if the CPU request rate has a weight of 0.3, the memory request rate has a weight of 0.3, and the GPU request rate has a weight of 0.4, then one formula for calculating the overall request rate is: Overall Request Rate = 0.3 × CPU Request Rate + 0.3 × Memory Request Rate + 0.4 × GPU Request Rate. This weighting logic takes into account both the basic resource guarantees of CPU and memory and highlights the core priority of GPU resources, ensuring that the evaluation results closely match the actual needs of the training task. However, the weights of each part are not limited to this; the above weight configuration is just an example.

[0106] Step 53: Calculate the scheduling score of each node based on the overall application rate, and prioritize scheduling the container groups corresponding to each working node to the nodes with higher scheduling scores; the lower the overall application rate, the higher the scheduling score.

[0107] The scheduler uses a scoring rule that is inversely proportional to the overall request rate and the scheduling score: the lower the overall request rate of a node, the smaller its resource occupancy ratio and the greater its idle potential, and the higher its scheduling score. For example, when the overall request rate is 0, the scheduling score is the full score of 10 points; when the overall request rate is 1, the scheduling score is 0 points.

[0108] During the scheduling phase, nodes with high scheduling scores are prioritized for container group deployment. This encourages multiple Worker nodes for similar training tasks to be deployed on a few nodes with available resources. This avoids resource fragmentation caused by multiple nodes being scattered across them, ensuring that large-scale training tasks requiring complete resources can be scheduled smoothly. At the same time, it improves the centralized utilization rate of node resources and reduces the risk of cross-node communication.

[0109] This method calculates the resource request rates of nodes' CPU, memory, and GPU, rather than simply relying on the absolute amount of resource requests. It guides container groups to centrally schedule tasks to nodes with low overall request rates, preventing tasks from being scattered across multiple nodes and reserving full resources for subsequent large-scale distributed training tasks. By assigning higher scheduling scores to lower overall request rates, a clear scheduling priority is established, widening the score differences between different nodes and avoiding the scheduling ambiguity of traditional strategies. This method enables efficient reuse of node resources through centralized scheduling, while reducing cross-node communication risks, and can meet the requirements of multi-machine, multi-GPU distributed training for resource continuity and core computing power assurance.

[0110] Example 6: To enhance understanding, this embodiment describes a complete workflow for a K8S multi-machine, multi-GPU distributed training task, from user submission to execution and monitoring. Figure 4 The diagram shows the overall flowchart of a distributed training task submission and scheduling process, which is divided into six main stages that proceed step by step: 1. User task submission process Users submit training tasks through a visual interface (supporting low-code / no-code operation), which actually creates a custom resource called a Training Job CRD. This resource includes core configurations such as training image, resource requirements (GPU / CPU / memory), and task priority, thus completing the declarative submission of business requirements.

[0111] 2. K8SAPIServer Receiving Phase After receiving the Training Job CRD request submitted by the user, the K8SAPIServer completes the storage and persistence of the CRD and broadcasts the creation event of the CRD to the cluster; the training task Operator listens for this event through the Watch mechanism and triggers the subsequent task processing flow.

[0112] 3. GPU Task Queue Management The GPU task queue manager intervenes: First, it calculates the priority of the current task based on user-defined priority, task waiting time, quota usage, and other dimensions. Then, it adds the task to the GPU queue of the corresponding priority and generates an estimated waiting time to be synchronized with the user. If there are waiting tasks in the high-priority queue and no idle GPU resources, it will trigger preemption logic, select a low-priority task to run, save its checkpoint file, release resources, allocate resources to the high-priority task, and at the same time, the preempted task will be promoted in priority and re-enqueued.

[0113] 4. Training Task Operator Processing Phase The training task operator parses the configuration information in the Training Job CRD and completes three core tasks: First, it matches the distributed architecture corresponding to the framework type (e.g., PyTorch corresponds to Worker mode, and TensorFlow corresponds to PS-Worker mode); second, it calculates the environment variables necessary for distributed training, such as RANK, WORLD_SIZE, and MASTER_ADDR; and third, it generates an affinity strategy to adapt to the framework (e.g., PyTorch task configuration for Pod affinity on the same node) and creates service resources (e.g., Service) to support communication between Workers.

[0114] 5. Customized scheduler scheduling process The customized scheduler performs precise scheduling: First, it performs Gang scheduling checks to ensure that the cluster resources meet the resource requirements of all Workers in the task; then it calculates the CPU, memory, and GPU resource request rates of each node and generates a weighted node scheduling score; after filtering nodes with affinity policies, it prioritizes scheduling Worker container groups to nodes with high scheduling scores; finally, it completes the scheduling binding of container groups.

[0115] 6. Training Execution and Monitoring After scheduling is complete, the cluster creates Worker Pods and injects preset environment variables, starts the distributed training process and initializes communication between Workers (such as Ring All Reduce); at the same time, it starts monitoring of resources and training metrics. If abnormal resource utilization is detected (such as low GPU utilization), it automatically performs adaptive resource adjustment (scaling up and down, GPU sharing); if a task fails, the task manager automatically triggers the replacement logic to ensure the continuous execution of training tasks.

[0116] Example 7: This embodiment relates to a K8S multi-machine multi-GPU distributed training device. A schematic diagram of the K8S multi-machine multi-GPU distributed training device in this embodiment can be seen as follows: Figure 5 As shown, it includes: a task monitoring module 201, a configuration extraction module 202, an architecture matching module 203, a runtime configuration generation module 204, a resource scheduling module 205, and a task collaborative execution module 206.

[0117] Among them, the task monitoring module 201 is used to continuously monitor the creation event of custom training tasks in the K8S API server; when the creation event is detected, the configuration extraction module is triggered. The configuration extraction module 202 is used to parse the fields in the custom training task and extract the user-defined training configuration information. Architecture matching module 203 is used to match the corresponding distributed training architecture based on the framework type in the training configuration information; Run configuration generation module 204 is used to generate run configuration information for distributed training adapted to the distributed architecture based on training configuration information and distributed training architecture. The resource scheduling module 205 is used to submit training configuration information and running configuration information to the customized scheduler to perform resource scheduling and create container groups and service resources corresponding to worker nodes; The task collaboration execution module 206 is used to control each working node to collaboratively execute training tasks based on the running configuration information after each container group starts up.

[0118] In the Kubernetes (K8S) multi-machine, multi-GPU distributed training device provided in this embodiment, the complex underlying K8S resource configuration is abstracted into high-level business requirements by relying on the linkage of the task monitoring module and the configuration extraction module. Users do not need to master underlying concepts such as Pods and Services; they only need to declare core information such as framework type and resource requirements through custom training tasks. The device can automatically complete the parsing and extraction, completely eliminating the complex manual configuration process in traditional solutions and significantly reducing the threshold for using distributed training. The architecture matching module can automatically match the optimal distributed architecture based on the framework type in the training configuration, without requiring manual specification by the user, and adapting to the communication characteristics and operational requirements of different frameworks. The runtime configuration generation module automatically calculates and generates runtime parameters adapted to the architecture based on the training configuration and the matched distributed architecture, avoiding the errors and inconsistencies that are prone to occur in manual configuration, and ensuring the consistency of the distributed training environment and the reliability of startup.

[0119] From task listening and triggering, configuration extraction, architecture matching, and runtime configuration generation to resource scheduling and task collaborative execution, the device forms a seamless control loop with each module. It eliminates the need for manual intervention in intermediate links, enabling one-click submission and automatic execution of distributed training tasks, which can significantly improve the deployment efficiency and maintenance convenience of training tasks.

[0120] Additionally, it should be noted that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units.

[0121] for example Figure 6 The diagram shows a module partitioning diagram of another K8S multi-machine multi-GPU distributed training device. In the diagram, the K8S multi-machine multi-GPU distributed training device is called the K8S control plane. The device mainly includes: training task CRD, GPU task queue manager, training task operator, and customized scheduler.

[0122] Among them, the training task CRD serves as the carrier for users to declare their requirements for distributed training tasks. It is used to define the high-level configuration fields of the training task (such as framework type, resource requirements, number of workers, priority, etc.), abstracting the user's business requirements for the training task into custom resources that can be recognized by K8S. It is the source of the basic configuration for subsequent task parsing, scheduling and execution.

[0123] The GPU Task Queue Manager is used to implement queuing and priority management of GPU training tasks. Specifically, it is responsible for core logic such as determining the GPU queuing activation status, calculating task priority (integrating user-set priority, waiting time and other dimensions), managing multi-priority queues, monitoring GPU resource idle status, and preempting and scheduling high-priority tasks, saving checkpoints of preempted tasks and adding them to the queue with increased weight, so as to solve the problem of disordered waiting of tasks in GPU resource-scarce scenarios.

[0124] The Training Task Operator acts as the controller for the Training Task CRD. It listens for the creation / update events of the Training Task CRD, automatically parses the configuration fields in the CRD to extract training requirements, matches the distributed training architecture corresponding to the framework type, generates runtime configuration information (such as environment variables, communication parameters, etc.) for the adapted architecture, and triggers the subsequent resource scheduling process. It is the core execution component that connects user requirements with underlying resource configuration.

[0125] The customized scheduler is used to schedule and deploy container groups and service resources. Specifically, it is responsible for calculating the CPU, memory, and GPU resource request rates of each node, weighting the overall request rate, and filtering nodes by combining the affinity strategy generated by the framework characteristics. It prioritizes scheduling container groups to nodes with low resource request rates, which avoids resource fragmentation, reserves complete resources for large-scale tasks, and optimizes worker distribution to reduce communication latency. It is a key scheduling component to ensure efficient resource utilization and training performance.

[0126] Furthermore, in order to highlight the innovative aspects of this application, no units that are not closely related to solving the technical problems proposed in this application are introduced in this embodiment, but this does not mean that there are no other units in this embodiment.

[0127] Example 8: Another embodiment of this application relates to an electronic device, such as... Figure 7 As shown, it includes: at least one processor 301; and a memory 302 communicatively connected to at least one processor 301; wherein the memory 302 stores instructions that can be executed by at least one processor 301, and the instructions are executed by at least one processor 301 to enable at least one processor 301 to perform the steps of the K8S multi-machine multi-card distributed training method in the above embodiments.

[0128] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0129] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0130] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A distributed training method for Kubernetes (K8S) multi-machine, multi-GPU systems, characterized in that: include: Continuously monitor the creation event of the custom training task in the Kubernetes API server; When the creation event is detected, the fields in the custom training task are parsed to extract the user-defined training configuration information; Match the corresponding distributed training architecture based on the framework type in the training configuration information; Based on the training configuration information and the distributed training architecture, generate distributed training runtime configuration information adapted to the distributed architecture; The training configuration information and the runtime configuration information are submitted to the customized scheduler for resource scheduling and the creation of container groups and service resources corresponding to the worker nodes. After each container group is started, it controls each of the working nodes to collaboratively execute training tasks based on the running configuration information.

2. The K8S multi-machine multi-card distributed training method according to claim 1, characterized in that, After controlling each of the aforementioned working nodes to collaboratively execute the training task based on the operational configuration information, the method further includes: Collect resource usage data of the container group; Based on the resource usage data, the ratio of the actual resource usage to the configured resource request amount for each resource is calculated as the resource request rate. Determine whether the resource request rate has reached the expansion threshold; if so, perform resource expansion. Determine whether the resource request rate is lower than the reduction threshold. If it is, perform resource reduction.

3. The K8S multi-machine multi-card distributed training method according to claim 2, characterized in that, The execution methods for resource expansion or resource reduction include: Save the current training checkpoint file; The new resource configuration parameters are calculated based on the resource application rate and the preset adjustment rules. Based on the new resource configuration parameters, create a new container group instance; Once the new container group instance reaches the ready state, the training checkpoint file will be migrated to the new container group instance. Delete the original running container group instance and update the configuration of the corresponding service resources so that the service resources point to the new container group instance.

4. The K8S multi-machine multi-card distributed training method according to claim 1, characterized in that, Before submitting the training configuration information and the runtime configuration information to the customized scheduler for resource scheduling and the creation of container groups and service resources corresponding to worker nodes, the process also includes: Read the GPU queuing enable configuration from the training configuration information; If the enabled configuration is enabled, the priority calculation algorithm is invoked to calculate the priority result in combination with the task information; the task information includes: user-set priority, task waiting time, user quota usage and task type. The training task is added to the GPU task queue corresponding to the priority based on the priority result; Monitor the idle status of the GPU resources and the waiting status of training tasks in each queue; When there are available GPU resources, the first schedulable task is selected from the corresponding queue in descending order of priority and scheduled to be executed on the available GPU resources.

5. The K8S multi-machine multi-card distributed training method according to claim 4, characterized in that, After monitoring the idle state of GPU resources and the waiting state of training tasks in each queue, the following is also included: If there are tasks waiting in the high-priority queue and no free GPU resources, check the tasks being executed in the low-priority queue and select the target task from low to high priority. After saving the checkpoint file of the target task, stop the target task and release the resources, and schedule high-priority waiting tasks to be executed on the released resources; The target task is then prioritized and added back to the original queue.

6. The K8S multi-machine multi-card distributed training method according to claim 4, characterized in that, Also includes: Based on the remaining time of the currently running task, the total duration of the preceding tasks in the queue, and the GPU resource release rate, the estimated waiting time of each task is calculated and generated, and added to the GPU task queue.

7. The K8S multi-machine multi-card distributed training method according to claim 1, characterized in that, The method of controlling each of the working nodes to collaboratively execute the training task based on the runtime configuration information further includes: Detect the GPU usage status of each container group and calculate the GPU utilization. When the GPU utilization rate is lower than a preset threshold, GPU sharing configuration is executed so that multiple training tasks can share the same GPU resources.

8. The K8S multi-machine multi-card distributed training method according to claim 1, characterized in that, The training configuration information and the runtime configuration information are submitted to the customized scheduler for resource scheduling and the creation of container groups and service resources corresponding to worker nodes, including: The customized scheduler calculates the CPU request rate, memory request rate, and GPU request rate of each node; The weighted overall request rate of the computing nodes is calculated based on the CPU request rate, the memory request rate, and the GPU request rate. The scheduling score for each node is calculated based on the overall request rate, and the container groups corresponding to each working node are preferentially scheduled to the nodes with higher scheduling scores; the lower the overall request rate, the higher the scheduling score.

9. A K8S multi-machine, multi-card distributed training device, characterized in that, include: The task monitoring module is used to continuously monitor the creation events of custom training tasks in the Kubernetes API server; When the creation event is detected, the configuration extraction module is triggered; The configuration extraction module is used to parse the fields in the custom training task and extract user-defined training configuration information. The architecture matching module is used to match the corresponding distributed training architecture based on the framework type in the training configuration information. The runtime configuration generation module is used to generate runtime configuration information for distributed training that is adapted to the distributed architecture, based on the training configuration information and the distributed training architecture. The resource scheduling module is used to submit the training configuration information and the running configuration information to the customized scheduler to perform resource scheduling and create container groups and service resources corresponding to the worker nodes; The task coordination execution module is used to control each of the working nodes to coordinately execute training tasks based on the running configuration information after each of the container groups is started.

10. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the K8S multi-machine multi-card distributed training method as described in any one of claims 1 to 8.

Images