Distributed AI job collective communication optimization method and device based on kubernetes

Abstract

The invention provides a distributed AI job collective communication optimization method and device based on kubernetes. The distributed AI job collective communication optimization method based on kubernetes comprises the steps that GPU simulation and proxy plug-ins are designed in a host, and an AI topology operator is designed on a kubernetes platform layer. After a kubernetes scheduler schedules a plurality of computing unit resource requests of a distributed AI task to a computing node, an AI topology operator constructs a resource topology for a container where each computing unit resource request on the same computing node is located, GPU and RDMA network card resource topologies of the distributed AI task are generated, a GPU simulation and proxy plug-in constructs a virtual GPU device / dev / vgpu according to the resource topologies, and the virtual GPU device / dev / vgpu is used for simulating the distributed AI task according to the virtual GPU device / dev / vgpu. The virtual GPU device may utilize a host NVLink or PCIe high-speed channel for data transfer and control communications through IO operation event forwarding and execution feedback. Through combination of GPU simulation and proxy plug-ins and an AI topology operator, closed-loop management is formed in the aspects of resource allocation, scheduling, virtual GPU equipment management and the like for a distributed AI job scene, so that the problem that distributed jobs in the same group cannot use high-speed NVLink or PCIe communication in the same host is solved, and unified affinity allocation of GPU and RDMA resources is realized.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a distributed AI job collective communication optimization method based on Kubernetes, a distributed AI job collective communication optimization device based on Kubernetes, an electronic device and a computer-readable medium. Background Art

[0002] Currently, AI large model training generally adopts distributed training. Distributed training is to build a cluster with super computing power and video memory capacity through multiple nodes to deal with the two main challenges of computing power wall and storage wall in large model training. The high-performance network connecting this super cluster directly determines the communication efficiency between intelligent computing nodes, which in turn affects the throughput and performance of the entire intelligent computing cluster. In order for the entire intelligent computing cluster to achieve high throughput, the high-performance network needs to have several key capabilities such as low latency, large bandwidth, long-term stability, large-scale scalability and operability.

[0003] To achieve distributed training of large models, at the physical layer, 8 GPU cards can be fully configured and 8 PCIe network card slots can be reserved. The GPUs in the node are connected through PCIe and NVLink, and the Spine-Leaf switching structure is used to connect across nodes. Based on the typical intelligent computing network topology, two intelligent computing nodes with the same GPU number across machines can communicate with each other in just one hop. Between intelligent computing nodes with different GPU numbers, with the help of Rail Local technology in the NCCL communication library, the bandwidth of NVSwitch between GPUs in the host can be fully utilized to convert the cross-card number communication between multiple machines into the same GPU card number communication between machines. In the current hardware architecture, the communication bandwidth between GPUs in the same host can reach 900GB / s through NVLink 4.0, the PCIe 5.0 bandwidth can reach 512GB / s, and the Mellanox CX7 RDMA network card bandwidth is 400Gb / s (50GB / s).

[0004] Nvidia's NCCL is one of the most widely used communication libraries, which implements multi-GPU and multi-node communication primitives optimized for GPUs and networks. The communication library provides routines such as all-gather, all-reduce, broadcast, reduce, reduce-scatter, and point-to-point send and receive, which are optimized to achieve high bandwidth and low latency through PCIe and NVLink high-speed interconnects within nodes and through RDMA networks between nodes.

[0005] Reference Figure 1Schematic diagram of creating distributed AI jobs in k8s. When using kubernetes to run AI distributed job tasks, a group of distributed job tasks will be scheduled to multiple nodes according to resource requirements. Multiple tasks may be assigned to a node. The communication between tasks is carried out through a network card with RDMA function. There is basically no difference in the communication efficiency between tasks on the same host and across hosts. The computing efficiency of large-scale AI distributed jobs is highly sensitive to computing and network performance. GPU and RDMA resources have an affinity relationship in physical structure. In the current kubernetes resource allocation system, GPU resources and RDMA network card resource allocation belong to independent resource types. Therefore, affinity optimal allocation cannot be achieved during resource scheduling and allocation.

[0006] Specifically, refer to Figure 2 The schematic diagram of AI distributed task communication based on k8s. The shortcomings of the existing AI distributed task communication method based on k8s and the reasons for the shortcomings are as follows:

[0007] ① When Kubernetes schedules multiple ranks of a distributed job to the same host, the communication between ranks cannot perceive the topology of Nvlink and PCIe due to the resource isolation of the container, and cannot use high-speed communication links. It can only use network card-level RDMA for communication, which will greatly reduce the performance of distributed jobs. (e.g. Figure 2 Communication between Pod1 and Pod2 in

[0008] ② The communication between multiple ranks and different GPU number tasks cannot use the Rail Local technology in the NCCL communication library, which will cause most of the cross-machine traffic to pass through the Spine switch. The traffic forwarding between multiple switches will cause certain performance loss. (e.g. Figure 2 The communication path between Pod3 and Pod5 in

[0009] ③GPU and RDMA resources cannot be uniformly allocated, which has a certain impact on the communication efficiency and network performance between multiple tasks. (e.g. Figure 2 Pod6 in the. Summary of the invention

[0010] In view of the above problems, the present invention is proposed to provide a Kubernetes-based distributed AI job collective communication optimization method and a corresponding Kubernetes-based distributed AI job collective communication optimization device, an electronic device and a computer-readable medium that overcome the above problems or at least partially solve the above problems.

[0011] The present invention discloses a distributed AI job collective communication optimization method based on Kubernetes, wherein a GPU simulation and proxy plug-in is provided in a host, and an AI topology operator is provided in a Kubernetes platform layer, and the method comprises:

[0012] After the kubernetes scheduler schedules multiple computing unit resource requests of the distributed AI task to the computing node, the AI ​​topology operator builds GPU and RDMA network card resource topologies for the container where each computing unit resource request is located on the same computing node, and generates the GPU and RDMA network card resource topologies of the distributed AI task; the computing unit resources include GPU resources and RDMA network card resources;

[0013] The GPU simulation and proxy plug-in builds a virtual GPU device / dev / vgpu according to the GPU and RDMA network card resource topology of the distributed AI task. The virtual GPU device can use the host NVLink or PCIe high-speed channel for data transmission and control communication through IO operation event forwarding and execution feedback.

[0014] Optionally, each computing unit resource request is to be configured with a GPU device and an RDMA network card device.

[0015] After the Kubernetes scheduler schedules multiple computing unit resource requests of the distributed AI task to the computing node, the AI ​​topology operator builds GPU and RDMA network card resource topologies for the container where each computing unit resource request is located on the same computing node, and generates GPU and RDMA network card resource topologies for the distributed AI task; the computing unit resources include GPU resources and RDMA network card resources, including:

[0016] After the Kubernetes scheduler schedules multiple computing unit resource requests of the distributed AI task to the computing nodes, the AI ​​topology operator allocates a first GPU device and a first network interface card to the first container where the first computing unit resource request is located on the first computing node, and allocates a second GPU device and a second network interface card to the second container where the second computing unit resource request is located on the first computing node; the AI ​​topology operator allocates a first GPU device and a first network interface card to the third container where the third computing unit resource request is located on the second computing node, and allocates a second GPU device and a second network interface card to the fourth container where the fourth computing unit resource request is located on the second computing node.

[0017] Optionally, the GPU simulation and proxy plug-in builds a virtual GPU device / dev / vgpu according to the GPU and RDMA network card resource topology of the distributed AI task. The virtual GPU device can use the host NVLink or PCIe high-speed channel for data transmission and control communication through IO operation event forwarding and execution feedback, including:

[0018] The GPU simulation and proxy plug-in divides the physical GPU resources into multiple virtual GPU devices according to the GPU and RDMA network card resource topology of the distributed AI task, generates device descriptors corresponding to the virtual GPU devices, and establishes a connection link between the physical GPU device and the virtual GPU device; the connection link includes a physical NVLink and a virtual NvLink;

[0019] The GPU simulation and proxy plug-in monitors the IO operation events performed by the application on the device descriptor corresponding to the virtual GPU device, forwards the IO operation events to the corresponding physical GPU device for execution, and returns the response of the physical GPU device to the virtual GPU device.

[0020] Optionally, the communication topology variable NCCL_TOPO_FILE is used to describe that the GPU devices are connected as physical NVLinks and / or virtual NvLinks; the communication topology variable NCCL_TOPO_FILE is pre-injected into the container's running environment when creating a container and mounting the main GPU and the virtual GPUs of the same group of tasks at the same time.

[0021] Optionally, the method further comprises:

[0022] When monitoring the request of the distributed AI task in the container to call CUDA for inter-GPU data transmission and the CUDA driver NCCL to perform topology discovery and topology detection according to the communication topology variable NCCL_TOPO_FILE, the GPU simulation and proxy plug-in determines the data transmission path according to the communication topology information described by the communication topology variable NCCL_TOPO_FILE;

[0023] If the data transfer path contains a virtual NVLink connection, the GPU simulation and proxy plug-in converts the request for inter-GPU data transfer into the corresponding virtual NVLink transfer operation and completes the data transfer through the virtual NVLink connection;

[0024] If the data transmission path is a physical NVLink connection, the GPU simulation and proxy plug-in complete the data transmission through the physical NVLink connection.

[0025] The present invention also discloses a distributed AI job collective communication optimization device based on Kubernetes, wherein a GPU simulation and proxy plug-in is provided in a host, and an AI topology operator is provided in a Kubernetes platform layer, and the device comprises:

[0026] A resource topology generation module, which is used for, after the kubernetes scheduler schedules multiple computing unit resource requests of the distributed AI task to the computing node, the AI ​​topology operator builds GPU and RDMA network card resource topology for the container where each computing unit resource request is located on the same computing node, and generates the GPU and RDMA network card resource topology of the distributed AI task; the computing unit resources include GPU resources and RDMA network card resources;

[0027] A virtual GPU construction and usage module is used for GPU simulation and proxy plug-ins to build a virtual GPU device / dev / vgpu according to the GPU and RDMA network card resource topology of the distributed AI task. The virtual GPU device can use the host NVLink or PCIe high-speed channel for data transmission and control communication through IO operation event forwarding and execution feedback.

[0028] Optionally, each computing unit resource request is to be configured with a GPU device and an RDMA network card device.

[0029] The resource topology generation module includes:

[0030] The resource allocation submodule is used to allocate a first GPU device and a first network interface card to the first container where the first computing unit resource request on the first computing node is located, and allocate a second GPU device and a second network interface card to the second container where the second computing unit resource request on the first computing node is located after the Kubernetes scheduler schedules multiple computing unit resource requests of the distributed AI task to the computing node. The AI ​​topology operator allocates a first GPU device and a first network interface card to the third container where the third computing unit resource request on the second computing node is located, and allocates a second GPU device and a second network interface card to the fourth container where the fourth computing unit resource request on the second computing node is located.

[0031] Optionally, the virtual GPU construction and use module includes:

[0032] A virtual GPU construction submodule, for GPU simulation and proxy plug-in to divide physical GPU resources into multiple virtual GPU devices according to the GPU and RDMA network card resource topology of the distributed AI task, generate device descriptors corresponding to the virtual GPU devices, and establish a connection link between the physical GPU device and the virtual GPU device; the connection link includes a physical NVLink and a virtual NvLink;

[0033] The virtual GPU uses a submodule for GPU simulation and proxy plug-in to monitor the IO operation events performed by the application on the device descriptor corresponding to the virtual GPU device, forward the IO operation events to the corresponding physical GPU device for execution, and return the response of the physical GPU device to the virtual GPU device.

[0034] Optionally, the communication topology variable NCCL_TOPO_FILE is used to describe that the GPU devices are connected as physical NVLinks and / or virtual NvLinks; the communication topology variable NCCL_TOPO_FILE is pre-injected into the container's running environment when creating a container and mounting the main GPU and the virtual GPUs of the same group of tasks at the same time.

[0035] Optionally, the device further comprises:

[0036] A communication topology detection module is used to monitor the request of the distributed AI task in the container to call CUDA for inter-GPU data transmission and the CUDA driver NCCL to perform topology discovery and topology detection according to the communication topology variable NCCL_TOPO_FILE. The GPU simulation and proxy plug-in determines the data transmission path according to the communication topology information described by the communication topology variable NCCL_TOPO_FILE;

[0037] A first data transmission module, configured to convert a request for data transmission between GPUs into a corresponding virtual NVLink transmission operation if a virtual NVLink connection is included in the data transmission path, and complete the data transmission through the virtual NVLink connection;

[0038] The second data transmission module is used for completing data transmission through the physical NVLink connection by the GPU simulation and proxy plug-in if the data transmission path is a physical NVLink connection.

[0039] The present invention also discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0040] The memory is used to store computer programs;

[0041] The processor is used to implement the distributed AI job collective communication optimization method based on Kubernetes as described in the present invention when executing the program stored in the memory.

[0042] The present invention also discloses one or more computer-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to execute the Kubernetes-based distributed AI job collective communication optimization method as described in the present invention.

[0043] The present invention includes the following advantages:

[0044] The present invention is based on the distributed AI job collective communication optimization method of kubernetes, designs GPU simulation and proxy plug-ins in the host, and designs AI topology operators at the kubernetes platform layer to realize topology perception and management for AI jobs. After the kubernetes scheduler schedules multiple computing unit resource requests of distributed AI tasks to the computing nodes, the AI ​​topology operator builds GPU and RDMA network card resource topologies for the containers where each computing unit resource request is located on the same computing node, generates GPU and RDMA network card resource topologies for distributed AI tasks, and GPU simulation and proxy plug-ins build virtual GPU devices / dev / vgpu according to the GPU and RDMA network card resource topologies of distributed AI tasks. The virtual GPU device can use the host NVLink or PCIe high-speed channel for data transmission and control communication through IO operation event forwarding and execution feedback. By combining GPU simulation and proxy plug-ins with AI topology operators, a closed-loop management is formed for distributed AI job scenarios in terms of resource allocation, scheduling, virtual GPU device management, etc., thereby solving the problem that the same group of distributed jobs cannot use high-speed NVLink or PCIe for communication within the same host, achieving unified affinity allocation of GPU and RDMA resources, and improving communication efficiency and network performance between multiple tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a schematic diagram of k8s creating a distributed AI job;

[0046] Figure 2 This is a schematic diagram of AI distributed task communication based on k8s;

[0047] Figure 3 It is a flowchart of the steps of a distributed AI job collective communication optimization method based on Kubernetes provided by an embodiment of the present invention;

[0048] Figure 4 It is a flow chart of optimizing the collective communication of distributed AI jobs based on Kubernetes provided by an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of distributed AI task resource scheduling and allocation provided by an embodiment of the present invention;

[0050] Figure 6 It is a schematic diagram of the workflow of the GPU simulation and proxy plug-in provided by an embodiment of the present invention;

[0051] Figure 7 It is a schematic diagram of the communication structure of a virtual GPU and a virtual NVlink provided by an embodiment of the present invention;

[0052] Figure 8 It is a structural block diagram of a distributed AI job collective communication optimization device based on Kubernetes provided by an embodiment of the present invention;

[0053] Fig. 9 is a block diagram of an electronic device provided by an embodiment of the present invention;

[0054] Fig.10 It is a schematic diagram of a computer-readable medium provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] Reference Figure 3 , shows a flowchart of a distributed AI job collective communication optimization method based on Kubernetes provided in an embodiment of the present invention, which may specifically include the following steps:

[0057] Step 301, after the kubernetes scheduler schedules multiple computing unit resource requests of the distributed AI task to the computing node, the AI ​​topology operator builds a GPU and RDMA network card resource topology for the container where each computing unit resource request is located on the same computing node, and generates a GPU and RDMA network card resource topology for the distributed AI task; the computing unit resources include GPU resources and RDMA network card resources;

[0058] Step 302, the GPU simulation and proxy plug-in builds a virtual GPU device / dev / vgpu according to the GPU and RDMA network card resource topology of the distributed AI task. The virtual GPU device can use the host NVLink or PCIe high-speed channel for data transmission and control communication through IO operation event forwarding and execution feedback.

[0059] In one embodiment of the present invention, each computing unit resource request is to be configured with a GPU device and an RDMA network card device.

[0060] After the Kubernetes scheduler schedules multiple computing unit resource requests of the distributed AI task to the computing node, the AI ​​topology operator builds GPU and RDMA network card resource topologies for the container where each computing unit resource request is located on the same computing node, and generates GPU and RDMA network card resource topologies for the distributed AI task; the computing unit resources include GPU resources and RDMA network card resources, including:

[0061] After the Kubernetes scheduler schedules multiple computing unit resource requests of the distributed AI task to the computing nodes, the AI ​​topology operator allocates a first GPU device and a first network interface card to the first container where the first computing unit resource request is located on the first computing node, and allocates a second GPU device and a second network interface card to the second container where the second computing unit resource request is located on the first computing node; the AI ​​topology operator allocates a first GPU device and a first network interface card to the third container where the third computing unit resource request is located on the second computing node, and allocates a second GPU device and a second network interface card to the fourth container where the fourth computing unit resource request is located on the second computing node.

[0062] In one embodiment of the present invention, the GPU simulation and proxy plug-in constructs a virtual GPU device / dev / vgpu according to the GPU and RDMA network card resource topology of the distributed AI task. The virtual GPU device can use the host NVLink or PCIe high-speed channel for data transmission and control communication through IO operation event forwarding and execution feedback, including:

[0063] The GPU simulation and proxy plug-in divides the physical GPU resources into multiple virtual GPU devices according to the GPU and RDMA network card resource topology of the distributed AI task, generates device descriptors corresponding to the virtual GPU devices, and establishes a connection link between the physical GPU device and the virtual GPU device; the connection link includes a physical NVLink and a virtual NvLink;

[0064] The GPU simulation and proxy plug-in monitors the IO operation events performed by the application on the device descriptor corresponding to the virtual GPU device, forwards the IO operation events to the corresponding physical GPU device for execution, and returns the response of the physical GPU device to the virtual GPU device.

[0065] In one embodiment of the present invention, the communication topology variable NCCL_TOPO_FILE is used to describe the physical NVLink connection and / or virtual NvLink connection between GPU devices; the communication topology variable NCCL_TOPO_FILE is pre-injected into the container's operating environment when creating a container and mounting the main GPU and the virtual GPU of the same group of tasks at the same time.

[0066] In one embodiment of the present invention, the method further comprises:

[0067] When monitoring the request of the distributed AI task in the container to call CUDA for inter-GPU data transmission and the CUDA driver NCCL to perform topology discovery and topology detection according to the communication topology variable NCCL_TOPO_FILE, the GPU simulation and proxy plug-in determines the data transmission path according to the communication topology information described by the communication topology variable NCCL_TOPO_FILE;

[0068] If the data transfer path contains a virtual NVLink connection, the GPU simulation and proxy plug-in converts the request for inter-GPU data transfer into the corresponding virtual NVLink transfer operation and completes the data transfer through the virtual NVLink connection;

[0069] If the data transmission path is a physical NVLink connection, the GPU simulation and proxy plug-in complete the data transmission through the physical NVLink connection.

[0070] The present invention provides a distributed AI job collective communication optimization method based on kubernetes, designs GPU simulation and proxy plug-ins in the host, can map the physical GPU to the Pod in the form of a virtual device through device simulation according to the topology of the AI ​​distributed job in the same host, and does not affect the mounting of the physical device to other Pods, and the CUDA and NCCL in the Pod can accurately build the GPU topology through the virtual device, including the driver layer can see the virtual GPU, the physical and virtual GPUs can communicate through high-speed NVLink or PCIe, when the rank in the Pod initiates cudaMemcpyPeer, the video memory data can be copied through NVLink; in the kubernetes platform layer, the AI ​​topology operator ai topology operator is designed to realize the topology perception and management of AI jobs, including the host GPU topology CRD and the topology management controller, and the vGPU Device Plugin can mount the virtual GPU device to the job Pod when creating the Pod; through the GPU simulation and proxy plug-in and the ai topology operator to form a closed-loop management in terms of resource allocation, scheduling, and virtual GPU device management for distributed AI job scenarios, thereby solving the problem that the same group of distributed jobs cannot use high-speed NVLink or PCIe for communication within the same host, and realizing a distributed AI job collective communication optimization method based on Kubernetes.

[0071] Reference Figure 4The following will introduce in detail the specific implementation process of the distributed AI job collective communication optimization method based on Kubernetes:

[0072] ① The user creates distributed tasks with GPU and RDMA resources in batches in Kubernetes. The resource requirement is 4 ranks, and each rank requires a GPU device and an RDMA network card device;

[0073] ②The kubernetes scheduler schedules rank0 and rank1 to node 1, and rank2 and rank3 to node 2 based on the node resource usage;

[0074] ③The ai topology operator builds GPU and RDMA network card resource topology for Pods assigned to the same host. On node 1, GPU0 and NIC0 are allocated to Pods where rank 0 is located, GPU1 and NIC1 are allocated to Pods where rank 1 is located, GPU0 and NIC0 are allocated to Pods where rank 2 is located on node 2, and GPU1 and NIC1 are allocated to Pods where rank 3 is located. For the topology diagram, refer to Figure 5 Add GPU and RDMA network card affinity topology management in the ai topology operator to achieve unified affinity allocation of GPU and RDMA resources, and further improve the communication efficiency between AI distributed job tasks.

[0075] ④vGPU Device Plugin starts the GPU proxy process according to the topology, refer to Figure 6 , GPU proxy plug-in is divided into two stages: initialization and proxy:

[0076] a. Initialization phase: constructing virtual GPU devices. The GPU proxy process first initializes and generates the corresponding / dev / vgpu0 and / dev / vgpu1 virtual GPU device descriptors and provides the connection links between the physical GPU and the virtual GPU, including the physical NVLink and vNvLink. It then monitors the IO operation events of other applications on the device descriptors.

[0077] b. Proxy working stage: When other applications perform IO operations on the device descriptor of the virtual GPU, the IO operations are forwarded to the corresponding physical GPU, and the response of the physical GPU is returned to the virtual GPU, thereby realizing the functions required by the virtual GPU.

[0078] ⑤ When creating a Pod, kubelet mounts the main GPU and the vGPU of the same group of tasks at the same time, and injects NCCL_TOPO_FILE into the running environment of the Pod for topology discovery of the communication library. The topology file description includes GPU devices and NVlink and vNVlink topology description information. The vGPUs of different groups of tasks are invisible to each other, which can achieve the effect of multi-tenant resource isolation.

[0079] When the AI ​​task in the Pod calls NCCL for topology discovery and detection, the communication topology can be built through the GPU device in the Pod.

[0080] The business communication process is specifically as follows:

[0081] Rank0 is bound to GPU0, and rank1 is bound to GPU1. When rank0 and rank1 need to transfer data between GPUs, CUDA's cudaMemcpyPeer is called to transfer video memory between GPUs. CUDA drives NCCL to perform initialization configuration, including communication topology discovery and topology detection. NCCL performs topology detection based on the environment variable NCCL_TOPO_FILE. NCCL_TOPO_FILE describes the physical NVlink connection and virtual vNVlink connection between GPU0 and GPU1. For the structural diagram, refer to Figure 7 .

[0082] When rank0 initiates the transfer of GPU memory data to rank1's GPU memory, NCCL can complete the construction of the communication topology through vGPU and vNvlink. In actual data transmission, NVLink is used to complete the data transmission.

[0083] The Kubernetes-based distributed AI job collective communication optimization method of the present invention optimizes the NCCL topology construction in the Pod through GPU simulation and proxy plug-ins, combined with the AI ​​distributed job business topology, and makes full use of the high-speed communication link NVLink in the host. It further unifies the affinity allocation of GPU and RDMA network card resources, greatly improving the communication efficiency of AI distributed training tasks.

[0084] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0085] Reference Figure 8 , shows a structural block diagram of a distributed AI job collective communication optimization device based on Kubernetes provided in an embodiment of the present invention, which may specifically include the following modules:

[0086] The resource topology generation module 801 is used for, after the kubernetes scheduler schedules multiple computing unit resource requests of the distributed AI task to the computing node, the AI ​​topology operator builds the GPU and RDMA network card resource topology for the container where each computing unit resource request is located on the same computing node, and generates the GPU and RDMA network card resource topology of the distributed AI task; the computing unit resources include GPU resources and RDMA network card resources;

[0087] The virtual GPU construction and use module 802 is used for GPU simulation and proxy plug-in to build a virtual GPU device / dev / vgpu according to the GPU and RDMA network card resource topology of the distributed AI task. The virtual GPU device can use the host NVLink or PCIe high-speed channel for data transmission and control communication through IO operation event forwarding and execution feedback.

[0088] Optionally, each computing unit resource request is to be configured with a GPU device and an RDMA network card device.

[0089] The resource topology generation module includes:

[0090] The resource allocation submodule is used to allocate a first GPU device and a first network interface card to the first container where the first computing unit resource request on the first computing node is located, and allocate a second GPU device and a second network interface card to the second container where the second computing unit resource request on the first computing node is located after the Kubernetes scheduler schedules multiple computing unit resource requests of the distributed AI task to the computing node. The AI ​​topology operator allocates a first GPU device and a first network interface card to the third container where the third computing unit resource request on the second computing node is located, and allocates a second GPU device and a second network interface card to the fourth container where the fourth computing unit resource request on the second computing node is located.

[0091] Optionally, the virtual GPU construction and use module includes:

[0092] A virtual GPU construction submodule, for GPU simulation and proxy plug-in to divide physical GPU resources into multiple virtual GPU devices according to the GPU and RDMA network card resource topology of the distributed AI task, generate device descriptors corresponding to the virtual GPU devices, and establish a connection link between the physical GPU device and the virtual GPU device; the connection link includes a physical NVLink and a virtual NvLink;

[0093] The virtual GPU uses a submodule for GPU simulation and proxy plug-in to monitor the IO operation events performed by the application on the device descriptor corresponding to the virtual GPU device, forward the IO operation events to the corresponding physical GPU device for execution, and return the response of the physical GPU device to the virtual GPU device.

[0094] Optionally, the communication topology variable NCCL_TOPO_FILE is used to describe that the GPU devices are connected as physical NVLinks and / or virtual NvLinks; the communication topology variable NCCL_TOPO_FILE is pre-injected into the container's running environment when creating a container and mounting the main GPU and the virtual GPUs of the same group of tasks at the same time.

[0095] Optionally, the device further comprises:

[0096] A communication topology detection module is used to monitor the request of the distributed AI task in the container to call CUDA for inter-GPU data transmission and the CUDA driver NCCL to perform topology discovery and topology detection according to the communication topology variable NCCL_TOPO_FILE. The GPU simulation and proxy plug-in determines the data transmission path according to the communication topology information described by the communication topology variable NCCL_TOPO_FILE;

[0097] A first data transmission module, configured to convert a request for data transmission between GPUs into a corresponding virtual NVLink transmission operation if a virtual NVLink connection is included in the data transmission path, and complete the data transmission through the virtual NVLink connection;

[0098] The second data transmission module is used for completing data transmission through the physical NVLink connection by the GPU simulation and proxy plug-in if the data transmission path is a physical NVLink connection.

[0099] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0100] In addition, an embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0101] Memory 903, used for storing computer programs;

[0102] The processor 901 is used to implement the distributed AI job collective communication optimization method based on Kubernetes as described in the above embodiment when executing the program stored in the memory 903.

[0103] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0104] The communication interface is used for communication between the above terminal and other devices.

[0105] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0106] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0107] like Fig.10 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 1001 is also provided, in which instructions are stored. When the computer-readable storage medium 1001 is run on a computer, the computer executes the distributed AI job collective communication optimization method based on Kubernetes described in the above embodiment.

[0108] In another embodiment provided by the present invention, a computer program product containing instructions is also provided. When the computer is run on a computer, the computer executes the Kubernetes-based distributed AI job collective communication optimization method described in the above embodiment.

[0109] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0110] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0111] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A distributed AI job collective communication optimization method based on kubernetes, characterized in that: A GPU simulation and proxy plug-in is provided in the host, and an AI topology operator is provided in the kubernetes platform layer. The method includes: After the kubernetes scheduler schedules multiple computing unit resource requests of the distributed AI task to the computing node, the AI ​​topology operator builds GPU and RDMA network card resource topologies for the container where each computing unit resource request is located on the same computing node, and generates the GPU and RDMA network card resource topologies of the distributed AI task; the computing unit resources include GPU resources and RDMA network card resources; The GPU simulation and proxy plug-in builds a virtual GPU device / dev / vgpu according to the GPU and RDMA network card resource topology of the distributed AI task. The virtual GPU device can use the host NVLink or PCIe high-speed channel for data transmission and control communication through IO operation event forwarding and execution feedback.

2. The method according to claim 1, characterized in that: Each computing unit resource request is to be configured with a GPU device and an RDMA network card device. After the kubernetes scheduler schedules multiple computing unit resource requests of the distributed AI task to the computing node, the AI ​​topology operator builds GPU and RDMA network card resource topology for the container where each computing unit resource request is located on the same computing node, and generates the GPU and RDMA network card resource topology of the distributed AI task; Computing unit resources include GPU resources and RDMA network card resources, including: After the Kubernetes scheduler schedules multiple computing unit resource requests of the distributed AI task to the computing nodes, the AI ​​topology operator allocates a first GPU device and a first network interface card to the first container where the first computing unit resource request is located on the first computing node, and allocates a second GPU device and a second network interface card to the second container where the second computing unit resource request is located on the first computing node; the AI ​​topology operator allocates a first GPU device and a first network interface card to the third container where the third computing unit resource request is located on the second computing node, and allocates a second GPU device and a second network interface card to the fourth container where the fourth computing unit resource request is located on the second computing node.

3. The method according to claim 1, characterized in that The GPU simulation and proxy plug-in builds a virtual GPU device / dev / vgpu based on the GPU and RDMA network card resource topology of the distributed AI task. The virtual GPU device can use the host NVLink or PCIe high-speed channel for data transmission and control communication through IO operation event forwarding and execution feedback, including: The GPU simulation and proxy plug-in divides the physical GPU resources into multiple virtual GPU devices according to the GPU and RDMA network card resource topology of the distributed AI task, generates device descriptors corresponding to the virtual GPU devices, and establishes a connection link between the physical GPU device and the virtual GPU device; the connection link includes a physical NVLink and a virtual NvLink; The GPU simulation and proxy plug-in monitors the IO operation events performed by the application on the device descriptor corresponding to the virtual GPU device, forwards the IO operation events to the corresponding physical GPU device for execution, and returns the response of the physical GPU device to the virtual GPU device.

4. The method according to claim 3, characterized in that: The communication topology variable NCCL_TOPO_FILE is used to describe whether the GPU devices are connected as physical NVLinks and / or virtual NvLinks. The communication topology variable NCCL_TOPO_FILE is pre-injected into the container's running environment when creating a container and mounting the main GPU and the virtual GPUs of the same task group at the same time.

5. The method according to claim 4, characterized in that The method further comprises: When monitoring the request of the distributed AI task in the container to call CUDA for inter-GPU data transmission and the CUDA driver NCCL to perform topology discovery and topology detection according to the communication topology variable NCCL_TOPO_FILE, the GPU simulation and proxy plug-in determines the data transmission path according to the communication topology information described by the communication topology variable NCCL_TOPO_FILE; If the data transfer path contains a virtual NVLink connection, the GPU simulation and proxy plug-in converts the request for inter-GPU data transfer into the corresponding virtual NVLink transfer operation and completes the data transfer through the virtual NVLink connection; If the data transmission path is a physical NVLink connection, the GPU simulation and proxy plug-in complete the data transmission through the physical NVLink connection.

6. A distributed AI job collective communication optimization device based on kubernetes, characterized in that: A GPU simulation and proxy plug-in is provided in the host, and an AI topology operator is provided at the kubernetes platform layer. The device includes: A resource topology generation module, which is used for, after the kubernetes scheduler schedules multiple computing unit resource requests of the distributed AI task to the computing node, the AI ​​topology operator builds GPU and RDMA network card resource topology for the container where each computing unit resource request is located on the same computing node, and generates the GPU and RDMA network card resource topology of the distributed AI task; the computing unit resources include GPU resources and RDMA network card resources; A virtual GPU construction and usage module is used for GPU simulation and proxy plug-ins to build a virtual GPU device / dev / vgpu according to the GPU and RDMA network card resource topology of the distributed AI task. The virtual GPU device can use the host NVLink or PCIe high-speed channel for data transmission and control communication through IO operation event forwarding and execution feedback.

7. The device according to claim 6, characterized in that Each computing unit resource request is to be configured with a GPU device and an RDMA network card device. The resource topology generation module includes: The resource allocation submodule is used to allocate a first GPU device and a first network interface card to the first container where the first computing unit resource request on the first computing node is located, and allocate a second GPU device and a second network interface card to the second container where the second computing unit resource request on the first computing node is located after the Kubernetes scheduler schedules multiple computing unit resource requests of the distributed AI task to the computing node. The AI ​​topology operator allocates a first GPU device and a first network interface card to the third container where the third computing unit resource request on the second computing node is located, and allocates a second GPU device and a second network interface card to the fourth container where the fourth computing unit resource request on the second computing node is located.

8. The device according to claim 1, characterized in that The virtual GPU construction and use module includes: A virtual GPU construction submodule, for GPU simulation and proxy plug-in to divide physical GPU resources into multiple virtual GPU devices according to the GPU and RDMA network card resource topology of the distributed AI task, generate device descriptors corresponding to the virtual GPU devices, and establish a connection link between the physical GPU device and the virtual GPU device; the connection link includes a physical NVLink and a virtual NvLink; The virtual GPU uses a submodule for GPU simulation and proxy plug-in to monitor the IO operation events performed by the application on the device descriptor corresponding to the virtual GPU device, forward the IO operation events to the corresponding physical GPU device for execution, and return the response of the physical GPU device to the virtual GPU device.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the Kubernetes-based distributed AI job collective communication optimization method as described in any one of claims 1 to 5 when executing the program stored in the memory.

10. One or more computer-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to execute the Kubernetes-based distributed AI job collective communication optimization method as described in any one of claims 1-5.

Images