Deployment method for realizing distributed training on k8s by using heterogeneous GPU (Graphics Processing Unit)

Abstract

The invention relates to the field of computers, and provides a deployment method for realizing distributed training on k8s by using a heterogeneous GPU (Graphics Processing Unit), which comprises the following steps of: compiling a distributed training task list file CR1 and submitting the file CR1 to the k8s; the first controller monitors the creation of the CR1, creates a plurality of CR2 according to the configuration of the CR1, and allocates training task parameters to each CR2; the controller II monitors the creation of each CR2, creates a plurality of Pods according to the configuration of each CR2, and allocates training task parameters to each Pod; the second controller judges whether current distributed training is elastic training or not, and if yes, an Hpa resource is created for each CR2; and after the current distributed training is completed, the controller I deletes all CR2 of the current distributed training task so as to delete all Pods. The technical problem that distributed training cannot be accelerated by simultaneously using multiple GPUs in one training task in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and specifically provides a deployment method for implementing distributed training on k8s using heterogeneous GPUs. Background Art

[0002] With the rapid development of cloud-native technologies represented by Kubernetes, more and more manufacturers are choosing to run AI training tasks on Kubernetes. However, as models become larger and larger, a single GPU can no longer load the entire model, and distributed training has become an inevitable trend. Therefore, how to quickly submit a distributed AI training task on Kubernetes in a simpler way has become a major problem.

[0003] Among the existing solutions, distributed AI training task deployment tools represented by kubeflow's training-operator solve this problem. The training-operator continuously monitors the creation of the CR resources it is concerned about, creates Pods using the Pod template defined in the CR, and configures different environment variables for the created Pods according to the parameters required by the training framework (such as Pytorch), marking them as belonging to a specific distributed training task, so that these Pods can complete the training together.

[0004] However, from the dominance of Nvidia GPUs to the large-scale deployment of various GPUs in actual production tasks, the AI ​​computing power of more data centers has maintained the coexistence of multiple GPUs. How to fully utilize these heterogeneous GPU resources to accelerate the training of a single large model has become a new problem. For example, a k8s cluster has m class A GPUs and n class B GPUs. When running a distributed training task in the k8s cluster, the current technology can only use m class A GPUs or n class B GPUs at most at the same time. No deployment system can quickly run a training task to use all m + n GPUs at the same time.

[0005] To use multiple heterogeneous GPUs in the same training task, you cannot use only one Pod template. This is because a Pod template can only configure one GPU and one container image, and in order to adapt to different GPUs, the container images are often different. In addition, when using Pod templates with different GPUs, the number of GPUs in a Pod generated by the template may also be different, so the CPU, memory and other resources required by a Pod may also be different. In the existing solution, kubeflow's training-operator only supports the configuration of one Pod template, and cannot configure multiple GPUs at the same time, so it is impossible to use multiple GPUs at the same time.

[0006] The existing technology kubeflow's training-operator realizes a simple and fast submission of a training task in k8s, but it only supports configuring one Pod template for one training task, and cannot fully utilize the heterogeneous GPU resources in the k8s cluster to accelerate training in the same training task.

[0007] Accordingly, the art needs a new deployment solution to solve the above problems. Summary of the invention

[0008] In order to overcome the above-mentioned defects, the present invention is proposed to solve the technical problem that the prior art cannot use multiple GPUs simultaneously in one training task to accelerate distributed training.

[0009] The present invention provides a deployment method for implementing distributed training using heterogeneous GPUs on k8s, comprising the steps of: Write the distributed training task list file CR1 according to the training framework requirements and submit it to k8s; Controller 1 monitors the creation of CR1, creates several CR2s according to the configuration of CR1, and assigns training task parameters to each CR2; Controller 2 monitors the creation of each CR2, creates several Pods according to the configuration of each CR2, and assigns training task parameters to each Pod; Controller 2 determines whether the current distributed training is elastic training. If so, it creates an Hpa resource for each CR2. After the current distributed training is completed, the controller deletes all CR2s of the current distributed training task to delete all Pods.

[0010] Further, the method further comprises the steps of: Controller 1 creates a headless service for the current distributed training, which is used for communication between Pods through domain names.

[0011] Further, the method further comprises the steps of: Controller 1 uses k8s informer to monitor updates of each CR2 and updates the status of each CR2 to CR1.Status. Controller 2 uses k8s informer to monitor updates of Pod and updates the status of each Pod to CR2.Status. CR1.Status represents the status of CR1, and CR2.Status represents the status of CR2.

[0012] Furthermore, CR1 includes a Pod.Spec field one and a list field, wherein the Pod.Spec field one is used to configure the fields supported by any k8s Pod template, and the list field is used to configure the GPU name, number of nodes, and image required for different types of GPUs in the training framework requirements.

[0013] Further, the controller 1 monitors the creation of CR1, creates a plurality of CR2s according to the configuration of CR1 and allocates training task parameters to each CR2, including the steps of: Controller 1 uses k8s informer to monitor the creation of CR1; Traverse the images, GPU names, and node numbers of different types of GPUs in the CR1 list field, and combine them with the Pod.Spec field of CR1 to form the Pod.Spec field 2 of CR2 corresponding to different types of GPUs. Controller 1 calculates the local range of NodeRank of each GPU based on the total number of nodes and the number of nodes of each GPU, determines the CPU and memory of each GPU, and updates them to the Pod.Spec field 2 of each CR2; Create each CR2 according to the generated Pod.Spec field 2.

[0014] Furthermore, the controller 2 monitors the creation of each CR2, creates a number of Pods according to the configuration of each CR2, and allocates training task parameters to each Pod, including the steps of: Controller 2 uses k8s informer to monitor the creation of each CR2; Controller 2 creates different Pod templates. The number of Pod templates is the same as that of CR2. Controller 2 creates Pod.Spec field 3 corresponding to each CR2 Pod based on the Pod template; Create each Pod according to the generated Pod.Spec field 3, and use the SetControllerReference method to set the Owner of each Pod to the corresponding CR2.

[0015] Furthermore, the controller 2 creates the Pod.Spec field 3 corresponding to each Pod of CR2 based on the Pod template, including: for each CR2, the controller 2 creates the Pod.Spec field 3 of each Pod based on the Pod template, assigns a NodeRank value to each Pod, determines the CPU and memory of each Pod, and updates them to the Pod.Spec field 3 of each Pod.

[0016] Further, the controller 2 determines whether the current distributed training is elastic training, including the controller 2 determining whether the number of nodes in CR2 is an interval value. If it is an interval value, the current distributed training is elastic training.

[0017] Working principle and beneficial effects of the present invention: In the technical solution for implementing the present invention, controller 1 creates several CR2s by parsing instance CR1, and controller 2 creates several Pods by parsing CR2. In a distributed training task, the parameters required for distributed training are allocated to each Pod through parameter distribution. Pods created using different templates are combined into the same training task, thereby achieving the purpose of using different GPUs simultaneously in the same training task. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In addition, similar numbers in the figures are used to represent similar components, among which: Figure 1 It is a flow chart of a deployment method for implementing distributed training using heterogeneous GPUs on k8s of the present invention; Figure 2 It is a flow chart of an implementation method of the distributed training deployment method of the present invention. DETAILED DESCRIPTION

[0019] Some embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0020] Figure 1 It is a flow chart of a deployment method for implementing distributed training on k8s using heterogeneous GPUs of the present invention. Figure 2 FIG. 1 is a flow chart of an implementation method of the distributed training deployment method of the present invention. Figure 1-2 As shown, a deployment method for implementing distributed training using heterogeneous GPUs on k8s in this embodiment mainly includes the following steps S1-S5.

[0021] S1: Write a distributed training task list file CR1 according to the training framework requirements, and use kubectl or other tools to submit CR1 to k8s; In one implementation, a distributed training task list file CR1 (i.e., custom resource) is written based on the training framework requirements, such as the need for all types of GPUs (graphics processors), the number of nodes and images required for each GPU, etc. CR1 includes a Pod.Spec field (i.e., container group template, hereinafter referred to as Pod.Spec field one) and a list field. Pod.Spec field one is used to configure the fields supported by any k8s Pod template; the list field is used to configure the different fields required by different types of GPUs in the training framework requirements, such as GPU name, number of nodes, image, etc.

[0022] In the present invention, the list field and the Pod.Spec field are defined in a field manner, and there is no need to configure a complete Pod template for each GPU, thereby simplifying the complexity of configuring the training task list.

[0023] S2: Controller 1 monitors the creation of CR1, creates several CR2s according to the configuration of CR1, and allocates training task parameters to each CR2.

[0024] In one implementation, the specific steps of step S2 include the following S21-S23.

[0025] S21: Controller 1 uses k8s informer to monitor the creation of CR1; S22: traverse the fields (image, GPU name, number of nodes) corresponding to different types of GPUs in the CR1 list field, and combine them with the Pod.Spec field of CR1 to form the Pod.Spec field (i.e., container group template, hereinafter referred to as Pod.Spec field 2) of CR2 (i.e., custom resource) corresponding to different types of GPUs.

[0026] This step corresponds to Figure 1 and Figure 2 The resulting "Pod.Spec field 2" corresponds to "GPU(s)+Pod.Spec" Figure 1 and Figure 2 "Pod.Spec" in the text box pointed by number arrows 4 and 5.

[0027] The image, GPU name, and number of nodes of each GPU are obtained through the list field. At the same time, the controller 1 calculates the local range of the NodeRank of each GPU based on the total number of nodes and the number of nodes of each GPU, determines the CPU and memory of each GPU, and updates them to the Pod.Spec field 2 of each CR2.

[0028] S23: Create each CR2 according to the generated Pod.Spec field 2.

[0029] S3: Controller 2 monitors the creation of each CR2, creates several Pods according to the configuration of each CR2, and assigns training task parameters to each Pod.

[0030] In one implementation, the specific steps of step S3 include the following S31-S33.

[0031] S31: Controller 2 uses k8s informer to monitor the creation of each CR2; S32: Controller 2 creates different Pod templates. The number of Pod templates is the same as that of CR2. S33: Controller 2 creates pods of each CR2 based on the Pod template; Specifically, for each CR2, controller 2 creates the Pod.Spec field (i.e., container group template, hereinafter referred to as Pod.Spec field 3) of each Pod based on the Pod template, assigns a NodeRank value to each Pod, determines the CPU and memory of each Pod, and updates them to the Pod.Spec field 3 of each Pod; Figure 2 In the example, "parameter1-1", "parameter1-2", "parameter2-1", and "parameter2-2" correspond to the parameter configuration in the third field of Pod.Spec, that is, the NodeRank value, CPU, and memory of each Pod. The local range of NodeRank (such as 0-1) is stored in the second field of Pod.Spec, while the specific value of 0 or 1 is stored in the third field of Pod.Spec.

[0032] S34: Create each Pod (i.e., container group) according to the generated Pod.Spec field 3, and use the SetControllerReference method to set the Owner of each Pod to the corresponding CR2.

[0033] The number of Pods created by each CR2 is the same as the number of GPU nodes corresponding to each CR2.

[0034] Furthermore, controller 1 creates a headless service for the current distributed training, which is used for Pod communication through domain names. In k8s, the IP address of the Pod is unstable and cannot be obtained before the Pod is started, while the domain name is stable and can be pre-fetched.

[0035] In this embodiment, controller one uses k8s informer to monitor the update of each CR2, and updates the status of each CR2 (the status update of CR2 is completed by controller two) to CR1.Status. Controller two uses k8s informer to monitor the update of Pod status, and updates the status of each Pod to CR2.Status. The update of Pod status is completed by k8s. CR1.Status represents the status of CR1, and CR2.Status represents the status of CR2.

[0036] S4: Controller 2 determines whether the current distributed training is elastic training. If so, it creates an Hpa resource for each CR2.

[0037] In one implementation, controller 2 determines whether the number of nodes of CR2 is an interval value. If it is an interval value, that is, the current distributed training is elastic training, an Hpa resource is created for CR2 so that it can automatically expand and shrink.

[0038] The number of nodes in CR2 is the same as the number of nodes in GPU, but the number of nodes in CR2 is separate from the second field of Pod.Spec. The number of nodes is not stored in the second field of Pod.Spec, but in a field in CR2.

[0039] S5: After the current distributed training is completed (successful or failed), the controller deletes all CR2s of the current distributed training task to delete all Pods.

[0040] In this embodiment, when creating a Pod, the SetControllerReference method is used to set the Owner of the Pod to CR2. By utilizing the cascading deletion capability of k8s, when CR2 is deleted, the Pod to which it belongs will also be automatically deleted to prevent completed or failed tasks from continuing to occupy resources.

[0041] Based on the above steps S1-S5, controller 1 creates several CR2s by parsing instance CR1, and controller 2 creates several Pods by parsing CR2. In a distributed training task, different Pod templates are constructed for different GPUs by parameter distribution, and Pods with the number of nodes corresponding to the GPU under the Pod template are created through different Pod templates. The parameters required for distributed training are assigned to each Pod, and Pods created using different templates are combined into the same training task, achieving the purpose of using different GPUs in the same training task at the same time. Then the overall state is reflected through the local state to obtain the final result of the training task, and finally the resources are recycled after the task is completed to form a closed loop.

[0042] In addition, in the present invention, controller 1 coordinates the parameter distribution of the entire training task, and controller 2 further distributes parameters and creates a training task Pod. The two-level controller mode can make the overall system more decoupled and facilitate the creation of Hpa for elastic training.

[0043] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order, and they can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0044] Here are some terms involved in the present invention.

[0045] K8s: also known as Kubernetes, abbreviated as k8s, is an open source container orchestration management tool.

[0046] Pod: The smallest control unit of k8s, used to carry and run actual applications. When running distributed training tasks in k8s, a Pod is a node for a training task.

[0047] Pod.Spec: Pod template, which can be used to create a Pod.

[0048] AI: Artificial Intelligence.

[0049] Nvidia: Nvidia, GPU manufacturer.

[0050] GPU: Graphics processor, mainly used for graphics rendering and parallel computing. In the field of AI, it is mainly used for parallel computing.

[0051] Kubeflow: A collection of tools for developing, training, optimizing, deploying, and managing machine learning on the k8s platform.

[0052] Training-operator: A component in kubeflow used to quickly deploy distributed training tasks on k8s.

[0053] CRD: Custom Resource Definition, used to extend k8s resources when k8s native resources are not sufficient to meet specific needs.

[0054] CR: custom resource, an instance of CRD. You can create multiple CRs for one CRD.

[0055] Operator: An application written to solve a k8s-oriented problem, usually composed of one or more controllers and one or more CRDs.

[0056] CR1, CR2: represent the instance of CRD1 corresponding to controller 1 and the instance of CRD2 corresponding to controller 2, respectively. CR1 is created by the user, and CR2 is created by controller 1.

[0057] CR1.Status, CR2.Status: indicate the status of CR1 and CR2 respectively. They are part of CR1 and CR2 and exist when CR1 and CR2 are created. They do not need to be created separately and are not assigned by the user. They are controlled by the controller and the user does not need to pay attention to them when creating them.

[0058] Informer: A core component in the k8s client library that monitors k8s API resource changes and notifies users when resources change.

[0059] Headless Service: A headless server used to provide a stable access entry for Pods.

[0060] Kubectl: A client command line tool for k8s.

[0061] Hpa: A component in k8s used to elastically expand and shrink Pods based on resources such as CPU and memory.

[0062] Pytorch: An open source deep learning framework known for its flexibility and ease of use.

[0063] NodeRank: A parameter assigned to each node in Pytorch distributed training, or to each Pod in k8s, which is a value between 0 and the number of nodes minus 1.

[0064] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A deployment method for implementing distributed training using heterogeneous GPUs on k8s, characterized in that: Includes steps: Write the distributed training task list file CR1 according to the training framework requirements and submit it to k8s; Controller 1 monitors the creation of CR1, creates several CR2s according to the configuration of CR1, and assigns training task parameters to each CR2; Controller 2 monitors the creation of each CR2, creates several Pods according to the configuration of each CR2, and assigns training task parameters to each Pod; Controller 2 determines whether the current distributed training is elastic training. If so, it creates an Hpa resource for each CR2. After the current distributed training is completed, the controller deletes all CR2s of the current distributed training task to delete all Pods.

2. According to claim 1, a deployment method for implementing distributed training using heterogeneous GPUs on k8s is characterized in that: Also includes the steps: Controller 1 creates a headless service for the current distributed training, which is used for communication between Pods through domain names.

3. According to claim 1, a deployment method for implementing distributed training using heterogeneous GPUs on k8s is characterized in that: Also includes the steps: Controller 1 uses k8s informer to monitor updates of each CR2 and updates the status of each CR2 to CR1.Status. Controller 2 uses k8s informer to monitor updates of Pod and updates the status of each Pod to CR2.Status. CR1.Status represents the status of CR1, and CR2.Status represents the status of CR2.

4. According to claim 1, a deployment method for implementing distributed training using heterogeneous GPUs on k8s is characterized in that: CR1 includes a Pod.Spec field 1 and a list field. The Pod.Spec field 1 is used to configure the fields supported by any k8s Pod template, and the list field is used to configure the GPU name, number of nodes, and image required for different types of GPUs in the training framework requirements.

5. According to claim 4, a deployment method for implementing distributed training using heterogeneous GPUs on k8s is characterized in that: Controller 1 monitors the creation of CR1, creates several CR2s according to the configuration of CR1, and assigns training task parameters to each CR2, including the following steps: Controller 1 uses k8s informer to monitor the creation of CR1; Traverse the images, GPU names, and node numbers of different types of GPUs in the CR1 list field, and combine them with the Pod.Spec field of CR1 to form the Pod.Spec field 2 of CR2 corresponding to different types of GPUs. Controller 1 calculates the local range of NodeRank of each GPU based on the total number of nodes and the number of nodes of each GPU, determines the CPU and memory of each GPU, and updates them to the Pod.Spec field 2 of each CR2; Create each CR2 according to the generated Pod.Spec field 2.

6. According to claim 5, a deployment method for implementing distributed training using heterogeneous GPUs on k8s is characterized in that: Controller 2 monitors the creation of each CR2, creates several Pods according to the configuration of each CR2, and assigns training task parameters to each Pod, including the following steps: Controller 2 uses k8s informer to monitor the creation of each CR2; Controller 2 creates different Pod templates. The number of Pod templates is the same as that of CR2. Controller 2 creates Pod.Spec field 3 corresponding to each CR2 Pod based on the Pod template; Create each Pod according to the generated Pod.Spec field 3, and use the SetControllerReference method to set the Owner of each Pod to the corresponding CR2.

7. According to claim 6, a deployment method for implementing distributed training using heterogeneous GPUs on k8s is characterized in that: The controller 2 creates the Pod.Spec field 3 corresponding to each Pod of CR2 based on the Pod template, including: for each CR2, the controller 2 creates the Pod.Spec field 3 of each Pod based on the Pod template, assigns a NodeRank value to each Pod, determines the CPU and memory of each Pod, and updates them to the Pod.Spec field 3 of each Pod.

8. According to claim 1, a deployment method for implementing distributed training using heterogeneous GPUs on k8s is characterized in that: The controller 2 determines whether the current distributed training is elastic training, including the controller 2 determining whether the number of nodes in CR2 is an interval value. If it is an interval value, the current distributed training is elastic training.

Images