Optimal communication scheduling method and system for multi-machine and multi-card GPUs for K8s
By combining the GPU topology data between computing nodes and within nodes in the K8s environment for optimal GPU communication combination allocation, the problem of unoptimized GPU selection in the prior art is solved, and the performance and efficiency of AI training tasks are improved.
Patent Information
- Application Number
- CN202411187291.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2044-08-28
AI Technical Summary
The existing K8s scheduler fails to fully consider the communication topology information between GPUs in AI training tasks, resulting in unoptimized GPU selection, affecting the performance of AI training tasks.
The kube-scheduler scheduler allocates the computing nodes that meet the GPU resource requests to the container group pod, and combines the GPU topology data between the computing nodes and within the nodes to perform optimal GPU communication combination allocation, and writes metadata to guide the GPU allocation of the kubelet.
The GPU communication performance between pods is optimized, the computing power of the GPU cluster is fully utilized, and the training process of AI training tasks is accelerated.
Smart Images
Figure CN118708533B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a cloud computing platform, and in particular to a multi-machine multi-card GPU optimal communication scheduling method and system for K8s. Background Art
[0002] The architecture of the k8s default scheduler kube-scheduler is as follows Figure 1 As shown in the figure, when a user submits a container group pod to the k8s cluster, kube-scheduler will select a container group pod to start scheduling. Each time kube-scheduler schedules a container group pod, it is divided into two stages: scheduling cycle and binding cycle. The scheduling cycle selects a computing node for the pod, and the binding cycle binds the pod to a server computing node, that is, it starts running on this computing node. The scheduling cycle includes two filtering stages, PreFilter and Filter, to select a group of computing nodes suitable for the pod to run. It also includes three scoring stages, PreScore, Score, and Nomalize Score, as well as Reserve and Permit stages. Finally, a computing node with the highest score is selected for use by the container group pod. Coscheduling is mainly a batch scheduling plug-in developed in the Permit stage of kube-scheduler.
[0003] With the rapid development of cloud-native technologies represented by kubernetes, more and more AI training tasks are beginning to be uniformly orchestrated and managed through the k8s platform. In addition, with the advancement of computing hardware, multi-machine and multi-card applications are becoming more and more widespread in the field of AI. Multi-machine and multi-card applications exhibit complex inter-GPU connectivity, requiring the scheduler to consider GPU topology information and GPU communication requirements between AI training task pods when allocating GPU resources. In addition, the communication bandwidth between multiple cards on the GPU server and the communication bandwidth between GPU servers are related to the network connection method. The traditional kube-scheduler scheduler is not aware of the GPU topology information of the node, and the selection of GPUs during the scheduling process is relatively random. Selecting different GPU combinations has a great impact on the training speed of AI tasks. At present, there are the following difficulties in the optimal GPU selection and scheduling for GPU communication between pods in the k8s environment:
[0004] 1: The default kube-scheduler scheduler only considers the number of GPUs when scheduling pods. After scheduling the pod to a node that meets the pod's GPU resource request, the node's kubelet then allocates specific GPU resources. The GPU resource allocation process does not consider the connection method (communication requirements) between GPUs, but simply allocates the number of GPU resource requests that meet the requirements. In this way, the GPUs allocated to the pod may not be the optimal combination for communication.
[0005] 2: The default kube-scheduler scheduler always schedules pods one by one in serial. Kubelet also allocates GPUs to pods one by one, and fails to consider the GPU communication requirements between a group of pods. At this time, the GPUs allocated to a group of pods are random, which cannot meet the optimal combination of GPU communication, reducing the training performance of AI training tasks.
[0006] The Chinese invention patent application with application number CN201910894815.1 discloses a working method and device for deep learning training tasks. By allocating GPUs to various deep learning training task types according to the remaining GPU resources in a single server or multiple server nodes, it achieves the effect of taking into account multiple types of deep learning training tasks while ensuring GPU utilization. Among them, the method includes: obtaining deep learning training task parameters input by the user; determining the deep learning training task type from the task parameters, and the deep learning training task type includes: single-machine type and multi-machine type; selecting GPUs according to different strategies according to different deep learning training task types; selecting the CPU with the shortest communication distance with the GPU according to the location of the GPU to work. This solution focuses on the bare metal GPU allocation method for single-machine and multi-machine deep learning training tasks. Although it considers the impact of the GPU connection method within the node and the network connection method across nodes on GPU communication, it does not describe the optimal GPU selection method for GPU communication between processes in the case of multi-machine deep learning training tasks with multiple processes. It also does not consider the application in the k8s environment, nor does it consider the optimal GPU selection scheduling method for GPU communication between pods in the k8s environment.
[0007] The Chinese invention patent application with application number CN202010601888.X discloses an efficient GPU resource allocation optimization method and system, which includes calling the GPU allocation interface to obtain the GPU resources and data information required for GPU allocation; the data information includes the physical topology structure, the NUMA grouping structure and the job information. According to the GPU static topology map in the physical topology map, the GPU topology communication factor is determined; according to the NUMA grouping structure and the job information, the GPU fragmentation factor is determined by adding corrections when calculating the GPY fragments. The obtained communication factor and fragmentation factor are weighted to determine the objective function value; the optimal GPU resource allocation scheme is obtained when the objective function value is minimized. Based on this method, an optimization system is also proposed. The invention claims that the allocation of GPU resources can not only ensure the computing performance of the GPU, but also greatly reduce the generation of GPU resource fragmentation, ensuring that each scheduled job can use the optimal configuration of the currently available GPU resources. This solution focuses on efficient GPU resource allocation methods for jobs to be scheduled. Although the impact of GPU allocation on jobs to be scheduled is weighed by the GPU topology communication factor and the GPU fragmentation factor, it does not describe the optimal GPU selection method for GPU communication between processes when the jobs to be scheduled are multi-process, because deep learning tasks often require multiple processes to communicate with each other to accelerate training speed. In addition, it does not consider applications in the k8s environment, nor does it consider the optimal GPU selection scheduling method for GPU communication between pods in the k8s environment. Summary of the invention
[0008] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a multi-machine and multi-card GPU optimal communication scheduling method and system for K8s are provided. When the scheduler assigns a specific GPU to an AI training task, the physical connection and communication topology between GPUs are considered to optimize the GPU communication performance between pods, thereby giving full play to the computing power of the GPU cluster and accelerating the training process of the AI training task.
[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0010] A multi-machine multi-card GPU optimal communication scheduling method for K8s includes the following steps:
[0011] After the user starts the AI training task including the container group pod, the kube-scheduler scheduler first allocates the computing nodes that meet the GPU resource request to each container group pod, and then combines the GPU topology data between the computing nodes with the GPU topology data within the computing node to perform the optimal GPU communication combination allocation, and writes the GPU allocation result into the metadata of the container group pod for storage. The optimal GPU communication combination allocation is performed by combining the GPU topology data between the computing nodes with the GPU topology data within the computing node. The steps include: if there is a container group pod that has not completed the allocation of computing nodes, for each container group pod that has completed the allocation of computing nodes, select the GPU with the largest number of NV connections as the GPU allocation result of the container group pod; if all container group pods have completed the allocation of computing nodes, select a GPU with the best communication performance with the GPU allocated to the scheduled container group pod according to the GPU topology data in the node as the GPU allocation result of the container group pod;
[0012] When the kubelet component of each computing node detects that a container group is being scheduled, it directly allocates a GPU to the container group based on the GPU allocation result written into the metadata of the container group pod and starts the container group pod to perform the AI training task.
[0013] Furthermore, the GPU topology data between the computing nodes is obtained by reading pre-configured GPU topology data between the nodes.
[0014] Furthermore, the GPU topology data in the computing node is collected and reported by the resource reporting module Nrt-agent in the computing node calling the GPU driver interface of the node.
[0015] Furthermore, allocating a computing node that meets the GPU resource request to each container group pod is specifically implemented through a NodeResourceFit scheduling plug-in, and the NodeResourceFit scheduling plug-in is configured as the container packaging mode BinPack to schedule the container group pods in the container group podgroup to the same computing node until the GPU resources of the computing node are fully occupied, and then find the next computing node with the closest communication distance to the computing node to continue scheduling the remaining container group pods in the container group podgroup until all the container group pods in the container group podgroup are scheduled.
[0016] Furthermore, the optimal GPU communication combination allocation is performed by combining the GPU topology data between the computing nodes with the GPU topology data within the computing node, which is implemented by the Coscheduling scheduling plug-in, and the Coscheduling scheduling plug-in is used to perform the optimal GPU communication combination allocation for the container group pod in the container group group podgroup after entering the Permit stage by combining the GPU topology data between the computing nodes with the GPU topology data within the computing node, and write the GPU allocation result into the metadata of the container group pod for storage.
[0017] Furthermore, when allocating a computing node that satisfies the GPU resource request to each container group pod, after all container group pods are allocated, the container group pods in the container group group podgroup that are allocated to the same computing node are assigned the record map["podGroupName"]["nodeName"], and the container group pods in the container group group podgroup that are not allocated to the same computing node are assigned the record map["podGroupName"], where "podGroupName" represents the name of the container group group, and "nodeName" is the name of the computing node.
[0018] Further, the Coscheduling plug-in is used to perform optimal GPU communication combination allocation for the container group pod in the container group podgroup after the container group podgroup enters the Permit stage in combination with the GPU topology data between the computing nodes and the GPU topology data within the computing node, including:
[0019] Determine whether the record map["podGroupName"] of the container group pod exists. If not, it means that the container group pod under the container group group podgroup of the container group pod has not yet completed the allocation of computing nodes. Select a GPU with the largest number of NV connections from the computing node of the container group pod as the GPU allocation result of the container group pod, end and exit;
[0020] If the record map["podGroupName"] of the container group pod exists, then determine whether the record map["podGroupName"]["nodeName"] of the container group pod exists. If so, it means that the container group pod and the container group pod that has been scheduled under the container group group podgroup of the container group pod are located on the same computing node. According to the GPU topology data in the node, select a GPU with the best communication performance with the GPU allocated to the container group pod that has been scheduled under the container group group podgroup of the container group pod as the GPU allocation result of the container group pod, and then end and exit;
[0021] If the record map["podGroupName"]["nodeName"] of the container group pod does not exist, it means that the container group pod and the container group pod scheduled under the container group group of the container group pod are located on different computing nodes. According to the GPU topology data between nodes, select a GPU with the best communication performance with the GPU allocated to the container group pod already scheduled under the container group group podgroup of the container group pod as the GPU allocation result of the container group pod, end and exit.
[0022] Furthermore, the GPU with the best communication performance refers to a GPU with the lowest latency or the highest bandwidth.
[0023] The present invention also proposes a multi-machine multi-card GPU optimal communication scheduling system for K8s, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute any one of the multi-machine multi-card GPU optimal communication scheduling methods for K8s.
[0024] The present invention also proposes a computer-readable storage medium, which stores a computer program / instruction, and the computer program / instruction is programmed or configured to execute any one of the multi-machine and multi-card GPU optimal communication scheduling methods for K8s through a processor.
[0025] Compared with the prior art, the present invention has the following advantages:
[0026] The present invention first allocates a computing node that meets the GPU resource request to each container group pod through the kube-scheduler scheduler, and then combines the GPU topology data between the computing nodes with the GPU topology data within the computing node to perform the optimal GPU communication combination allocation, thereby completing the work of allocating a specific GPU to the AI training task in the kube-scheduler scheduling stage, and there is no need to allocate GPUs in the kubelet stage. When allocating a specific GPU to the AI training task, the physical connection and communication topology between GPUs are considered to optimize the GPU communication performance between the container group pods, thereby giving full play to the computing power of the GPU cluster and accelerating the training process of the AI training task. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 This is the architecture diagram of the k8s default scheduler kube-scheduler.
[0028] Figure 2 It is an overall design diagram of an embodiment of the present invention.
[0029] Figure 3 The following is a brief flow chart of the overall design of an embodiment of the present invention.
[0030] Figure 4 Schematic diagram of specific steps of an embodiment of the present invention.
[0031] Figure 5 This is an example of a GPU network communication topology diagram between computing nodes.
[0032] Figure 6 This is an example of the GPU topology diagram in a computing node. DETAILED DESCRIPTION
[0033] Before describing the specific embodiments of the present invention in detail, relevant technical terms are first explained.
[0034] K8s: also known as Kubernetes, abbreviated as k8s, is an open source container orchestration management tool.
[0035] Pod: The smallest control unit of k8s, which can contain one or more containers. It is usually directly translated as a "container group" and is used to carry and run actual applications. It is also the smallest scheduling unit in k8s. Pod can also be started and run independently without the need to be controlled by a controller in k8s.
[0036] Kubelet: It is an agent component on the kubernetes worker node. It runs on each worker node. Its main task is to allocate CPU, memory, GPU and other resources to Pod and to create and delete Pod.
[0037] NUMA: Non-uniform memory access (NUMA) is a memory architecture designed for multi-processor computers. It separates CPU resources and groups them into nodes. Each NUMA node has its own unique CPU, memory, GPU and other resources.
[0038] AI training tasks: refers to the use of machine learning or deep learning algorithms to train models so that the models can perform specific tasks or solve specific problems. In order to speed up the completion of training tasks and generate training models, AI training tasks are generally composed of 2 or more pods to jointly complete a training task.
[0039] Multiple machines and multiple cards: multiple server computing nodes, each with multiple GPU cards installed.
[0040] Connection method between GPUs: The connection method between GPUs affects the communication performance between GPUs. For NVIDIA graphics cards, NVLinks (abbreviated as NV) has the strongest communication performance. PIX: Communication within the same PCIeSwitch. PXB: Communication through multiple PCIe Switches without passing through the PCIe Host Bridge. PHB: Communication only through one PCIe Host Bridge. (This is the communication method between CPUs and GPUs in the same group). NODE: Communication within the NUMA group through different PCIe Host Bridges. SYS: Can only communicate across sockets, through QPI / UPI (that is, across NUMA groups). The communication performance of NV, PXB, PHB, NODE, and SYS decreases in turn.
[0041] Kube-scheduler: The default scheduler of k8s, a component in k8s used to schedule pods to a server node.
[0042] Permit: The late scheduling stage in the kube-scheduler scheduler is used to control whether the pod enters the subsequent scheduling stage, that is, to allow or reject the pod or let the pod wait to enter the subsequent scheduling stage. The present invention will construct and implement the optimal communication scheduling method for multiple machines and multiple GPUs in the permit stage.
[0043] PodGroup: A pod group consisting of a group of pods, which can define the total resource information such as the number of pods in the group and the number of GPUs required.
[0044] Coscheduling: An open source scheduling plug-in developed based on the scheduling framework of the kube-scheduler scheduler. It is mainly used in the field of batch scheduling. Since the kube-scheduler scheduler always schedules pods one by one in serial, in order to ensure that a group of pods can be scheduled at the same time, coscheduling is an extension of the permit phase of the kube-scheduler scheduler. When the current number of pods is less than the number of pods specified in the podgroup group, the pod waits in the permit phase until the current number of pods is greater than or equal to the number of pods specified in the podgroup group. All pods are allowed to enter the subsequent scheduling phase at one time, ensuring that a group of pods can be scheduled at the same time.
[0045] NodeResourceFit: An open source scheduling plug-in developed based on the scheduling framework of the kube-scheduler scheduler. It can mainly control the scheduling of pods to the same node as much as possible to reduce node resource fragmentation.
[0046] BinPack: A configuration mode of the NodeResourceFit scheduling plug-in that schedules pods to the same node as much as possible to reduce node resource fragmentation.
[0047] Nrt-agent: developed by the present invention, i.e., node resource topology agent component (node resource topologyagent), used to collect resource topology information such as CPU, memory, GPU, etc. in k8s cluster nodes. The present invention mainly uses the topology information of GPU in the node. The GPU topology information in the node includes GPU-id, GPU device name, and connection method between GPUs.
[0048] Embodiment 1
[0049] This embodiment proposes a multi-machine multi-card GPU optimal communication scheduling method for K8s, which constructs and implements multi-machine multi-card GPU optimal communication scheduling in the Permit stage, and selects the optimal GPU communication combination for a group of container groups pods in the container group cluster podgroup in the Permit stage, so as to achieve the optimal GPU communication between the group of pods and improve the training speed of AI training tasks. Figure 2 , Figure 3 and Figure 4 As shown, the following steps are included:
[0050] The pre-configuration phase includes:
[0051] Step 101, the user configures GPU topology data between nodes;
[0052] Step 102, deploy the resource reporting module Nrt-agent;
[0053] Step 103: deploy relevant scheduling plug-ins, which specifically include the NodeResourceFit scheduling plug-in and the Coscheduling scheduling plug-in of the kube-scheduler scheduler. The NodeResourceFit scheduling plug-in schedules the container group pods to the same node as much as possible to reduce node resource fragmentation. The Coscheduling scheduling plug-in is used to ensure that a group of pods can be scheduled at the same time in the Permit phase of the kube-scheduler scheduler, and complete the optimal GPU communication combination allocation of these pods (located in the same podgroup) in the Permit phase.
[0054] The GPU scheduling stage includes:
[0055] Step 104: The user starts an AI training task, and the scheduling plug-in performs filtering, scoring, permitting, and binding operations to complete the optimal GPU communication combination allocation;
[0056] The AI training task of this embodiment includes one or more container group pods located in the same container group cluster podgroup. After the user starts the AI training task including the container group pod, the container group pod in the AI training task first passes through the NodeResourceFit scheduling plug-in of the kube-scheduler scheduler to allocate computing nodes that meet the GPU resource request to each container group pod;
[0057] The container group pod then passes through the Coscheduling plug-in of the kube-scheduler scheduler. In the permit phase, the GPU topology data between computing nodes and the GPU topology data within the computing node are combined to perform optimal GPU communication combination allocation, and the GPU allocation result is written into the metadata of the container group pod for storage;
[0058] Step 105: When the kubelet component of each computing node detects that a container group pod is scheduled, it directly allocates a GPU to the container group pod according to the GPU allocation result written into the metadata of the container group pod and starts the container group pod to execute the AI training task.
[0059] Each step is described in detail below.
[0060] In step 101 of this embodiment, the GPU topology data between nodes is specifically configured by the user according to the network connection status between the cluster nodes, for example, according to Figure 5The inter-node GPU network communication topology diagram shown can be used to derive the GPU topology data between nodes as shown in the following table:
[0061] Table 1 GPU topology data between nodes
[0062]
[0063] Correspondingly, in step 104, the GPU topology data between the computing nodes is obtained by reading the pre-configured GPU topology data between the nodes.
[0064] In step 102 of this embodiment, the resource reporting module Nrt-agent is deployed in all computing nodes of the cluster, and generates GPU topology data in the node by obtaining the hardware resource information of the cluster node. Specifically, Nrt-agent calls the node-related GPU driver interface to find the number of GPUs in the node, GPU-id, GPU type, and the connection method between GPUs, and finally generates GPU topology data in the node. For example, according to Figure 6 The GPU topology diagram within the node shown can be used to obtain the GPU topology data within the node as shown in the following table:
[0065] Table 2 GPU topology data within the node
[0066]
[0067] Correspondingly, in step 104, the GPU topology data in the computing node is collected and reported by the resource reporting module Nrt-agent in the computing node calling the GPU driver interface of the node.
[0068] In step 103 of this embodiment, when deploying the NodeResourceFit scheduling plug-in, the following steps are specifically performed:
[0069] The user deploys the NodeResourceFit scheduling plug-in of kube-scheduler, and then configures the NodeResourceFit scheduling plug-in as the container packaging BinPack mode. This can schedule a group of container group pods under the container group cluster podgroup to the same node as much as possible until the GPU resources of the node are fully occupied, reducing GPU resource fragmentation.
[0070] Correspondingly, in step 104, when the container group pod passes through the NodeResourceFit scheduling plug-in, since the NodeResourceFit scheduling plug-in is configured as the container packaging mode BinPack, the container group pods in the same container group podgroup are scheduled to the same computing node until the GPU resources of the computing node are fully occupied, and then the next computing node with the closest communication distance to the computing node is found to continue scheduling the remaining container group pods in the container group podgroup until all the container group pods in the container group podgroup are scheduled. Therefore, when a container group pod enters the Permit stage of Coscheduling, the container group pod already knows its specific scheduling node.
[0071] In step 104 of the present embodiment, the optimal GPU communication combination allocation is performed by combining the GPU topology data between the computing nodes with the GPU topology data within the computing node through the Coscheduling scheduling plug-in, and according to step 103, it can be known that the Coscheduling scheduling plug-in is used to perform the optimal GPU communication combination allocation for the container group pod in the container group group podgroup after entering the Permit stage by combining the GPU topology data between the computing nodes with the GPU topology data within the computing node, and write the GPU allocation result into the metadata of the container group pod for storage.
[0072] In this embodiment, when a computing node that satisfies the GPU resource request is allocated to each container group pod, after each container group pod is allocated, the container group pods in the container group group podgroup that are allocated to the same computing node are assigned the record map["podGroupName"]["nodeName"], and the container group pods in the container group group podgroup that are not allocated to the same computing node are assigned the record map["podGroupName"], where "podGroupName" represents the name of the container group group, and "nodeName" is the name of the computing node.
[0073] As mentioned above, when a container group pod enters the Permit stage of Coscheduling, the container group pod already knows the specific scheduling node. The Coscheduling scheduling plug-in is used to perform the optimal GPU communication combination allocation for the container group pod in the container group group podgroup after it enters the Permit stage, combining the GPU topology data between the computing nodes and the GPU topology data within the computing node, including:
[0074] Step 201, determine whether the record map["podGroupName"] of the container group pod exists. If not, it means that the container group pod under the container group group podgroup of the container group pod has not completed the allocation of computing nodes. Select a GPU with the largest number of NV connections from the computing nodes of the container group pod as the GPU allocation result of the container group pod, end and exit; if it exists, jump to step 202;
[0075] Step 202, determine whether the record map["podGroupName"]["nodeName"] of the container group pod exists. If so, it means that the container group pod and the container group pod scheduled under the container group group podgroup of the container group pod are located on the same computing node, and jump to step 203; if not, it means that the container group pod and the container group pod scheduled under the container group group of the container group pod are located on different computing nodes, and jump to step 204;
[0076] Step 203, according to the GPU topology data in the node, select a GPU with the best communication performance with the GPU allocated to the container group pod that has been scheduled under the container group group podgroup of the container group pod as the GPU allocation result of the container group pod. The GPU with the best communication performance refers to the GPU with the lowest latency or the highest bandwidth. The process ends and exits.
[0077] Step 204, based on the GPU topology data between nodes, select a GPU with the best communication performance with the GPU allocated to the container group pod that has been scheduled under the container group group podgroup of the container group pod as the GPU allocation result of the container group pod. The GPU with the best communication performance refers to the GPU with the lowest latency or the highest bandwidth. End and exit.
[0078] In step 104 of this embodiment, the GPU allocation result is written into the metadata of the container group pod for storage. Specifically, after the GPU allocation of each container group pod in the container group podgroup is completed, the GPU scheduling allocation result is written into the metadata of the corresponding container group pod in real time for storage until the group of container group pods in the container group podgroup is scheduled.
[0079] Embodiment 2
[0080] This embodiment proposes a multi-machine multi-card GPU optimal communication scheduling system for K8s, including a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the multi-machine multi-card GPU optimal communication scheduling method for K8s described in Embodiment 1, which includes:
[0081] The resource reporting module, namely Nrt-agent, calls the node-related GPU driver interface to discover the number of GPUs in the node, GPU-id, GPU type, and the connection method between GPUs, and finally generates the GPU topology data in the node;
[0082] The scheduling plug-in module, namely the Permit part in Coscheduling, is used to achieve the optimal GPU combination allocation for GPU communication.
[0083] The system workflow of this implementation is as follows:
[0084] 1. The user configures the GPU topology data between nodes and Nrt-agent reports the GPU topology data within the node;
[0085] 2. According to the GPU topology data between nodes and the GPU topology data within the node, a GPU combination with optimal communication is scheduled for a group of container groups in the container group cluster podgroup, while reducing resource fragmentation and optimizing the GPU communication performance between container group pods, thereby giving full play to the computing power of the GPU cluster and accelerating the training process of AI training tasks.
[0086] This embodiment also proposes a computer-readable storage medium, which stores a computer program / instruction. The computer program / instruction is programmed or configured to execute the multi-machine and multi-card GPU optimal communication scheduling method for K8s described in Example 1 through a processor.
[0087] This embodiment also proposes a computer program product, including a computer program / instruction, which is programmed or configured to execute the multi-machine and multi-card GPU optimal communication scheduling method for K8s described in Example 1 through a processor.
[0088] In summary, the present invention discloses a multi-machine multi-card GPU optimal communication scheduling method and system for K8s, a method and system for allocating a group of container groups pods of a container group cluster podgoup with an optimal communication GPU combination in a kubernetes environment, so that the GPUs allocated to the container group pods have the highest communication performance, while reducing GPU resource fragmentation and accelerating the training process of AI training tasks. In addition, the problem of random allocation of GPUs by the current kube-scheduler scheduler is solved, and the allocation of GPUs is unified by the kube-scheduler scheduler, and there is no need to allocate GPUs to kubelet, which reduces the GPU allocation call chain and improves the GPU allocation efficiency of the entire system.
[0089] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0090] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. An optimal communication scheduling method for multi-machine and multi-GPU for K8s, It is characterized in that The steps include: After the user starts an AI training task that includes a container group pod, the kube-scheduler scheduler first allocates a computing node that meets the GPU resource request to each container group pod, and then combines the GPU topology data between the computing nodes with the GPU topology data within the computing node to perform the optimal GPU communication combination allocation, and writes the GPU allocation result to the metadata of the container group pod for storage. The optimal GPU communication combination allocation based on the GPU topology data between the computing nodes and the GPU topology data within the computing node includes: if a container group pod has not completed the allocation of computing nodes, for each container group pod that has completed the allocation of computing nodes, select the GPU with the largest number of NV connections as the GPU allocation result of the container group pod; if all container group pods have completed the allocation of computing nodes and are located on the same computing node, according to GPU topology data within the node, select a GPU with the best communication performance with the GPU assigned to the scheduled container group pod as the GPU allocation result of the container group pod; if all container group pods have completed the allocation of computing nodes and are located on different computing nodes, select a GPU with the best communication performance with the GPU assigned to the scheduled container group pod as the GPU allocation result of the container group pod according to the GPU topology data between nodes; the GPU topology data between nodes is specifically the GPU topology data between nodes configured by the user according to the network connection between cluster nodes, the GPU topology data within the node includes GPU-id, GPU device name, and connection method between GPUs, and the GPU with the best communication performance refers to the GPU with the lowest latency or the highest bandwidth; When the kubelet component of each computing node detects that a container group pod is scheduled, it directly allocates a GPU to the container group based on the GPU allocation result written into the metadata of the container group pod and starts the container group pod to perform the AI training task.
2. According to the K8s-oriented multi-machine multi-card GPU optimal communication scheduling method of claim 1, It is characterized in that The GPU topology data between the computing nodes is obtained by reading the pre-configured GPU topology data between the nodes.
3. According to the K8s-oriented multi-machine multi-card GPU optimal communication scheduling method of claim 1, It is characterized in that The GPU topology data in the computing node is collected and reported by the resource reporting module Nrt-agent in the computing node calling the GPU driver interface of the node.
4. According to claim 1, the optimal communication scheduling method for multi-machine and multi-card GPUs for K8s, It is characterized in that Allocating a computing node that meets the GPU resource request for each container group pod is specifically implemented through the NodeResourceFit scheduling plug-in, and the NodeResourceFit scheduling plug-in is configured as the container packaging mode BinPack to schedule the container group pods in the container group podgroup to the same computing node until the GPU resources of the computing node are full, and then find the next computing node with the closest communication distance to the computing node to continue scheduling the remaining container group pods in the container group podgroup until all the container group pods in the container group podgroup are scheduled.
5. According to claim 1, the optimal communication scheduling method for multi-machine and multi-card GPUs for K8s, It is characterized in that The optimal GPU communication combination allocation is performed by combining the GPU topology data between the computing nodes with the GPU topology data within the computing node through the Coscheduling scheduling plug-in, and the Coscheduling scheduling plug-in is used to perform the optimal GPU communication combination allocation for the container group pod in the container group group podgroup after entering the Permit stage by combining the GPU topology data between the computing nodes with the GPU topology data within the computing node, and write the GPU allocation result into the metadata of the container group pod for storage.
6. The optimal communication scheduling method for multi-machine and multi-card GPUs for K8s according to claim 5, It is characterized in that When allocating a computing node that meets the GPU resource request to each container group pod, after all container group pods are allocated, the container group pods in the container group group podgroup that are allocated to the same computing node are assigned the record map["podGroupName"]["nodeName"], and the container group pods in the container group group podgroup that are not allocated to the same computing node are assigned the record map["podGroupName"], where "podGroupName" represents the name of the container group group, and "nodeName" is the name of the computing node.
7. The optimal communication scheduling method for multi-machine and multi-card GPUs for K8s according to claim 6, It is characterized in that The Coscheduling plug-in is used to perform optimal GPU communication combination allocation for the container group pod in the container group podgroup after the container group podgroup enters the Permit stage in combination with the GPU topology data between the computing nodes and the GPU topology data within the computing node, including: Determine whether the record map["podGroupName"] of the container group pod exists. If not, it means that the container group pod under the container group group podgroup of the container group pod has not yet completed the allocation of computing nodes. Select a GPU with the largest number of NV connections from the computing node of the container group pod as the GPU allocation result of the container group pod, end and exit; If the record map["podGroupName"] of the container group pod exists, determine whether the record map["podGroupName"]["nodeName"] of the container group pod exists. If so, it means that the container group pod and the container group pod scheduled under the container group group podgroup of the container group pod are located on the same computing node. According to the GPU topology data in the node, select a GPU with the best communication performance with the GPU allocated to the container group pod already scheduled under the container group group podgroup of the container group pod as the GPU allocation result of the container group pod, end and exit; If the record map["podGroupName"]["nodeName"] of the container group pod does not exist, it means that the container group pod and the container group pod scheduled under the container group group of the container group pod are located on different computing nodes. According to the GPU topology data between nodes, select a GPU with the best communication performance with the GPU allocated to the container group pod already scheduled under the container group group podgroup of the container group pod as the GPU allocation result of the container group pod, end and exit.
8. A multi-machine multi-card GPU optimal communication scheduling system for K8s, including interconnected microprocessors and memories, It is characterized in that The microprocessor is programmed or configured to execute the multi-machine and multi-card GPU optimal communication scheduling method for K8s as described in any one of claims 1 to 7.
9. A computer-readable storage medium having a computer program / instructions stored therein, It is characterized in that The computer program / instruction is programmed or configured to execute the optimal communication scheduling method for multi-machine and multi-card GPUs for K8s as described in any one of claims 1 to 7 through a processor.