Cross-node RDMA high-performance communication method in Kubernetes cluster

Abstract

The invention discloses a cross-node RDMA (Remote Direct Memory Access) high-performance communication method in a Kubernetes cluster, which belongs to the technical field of computing networks, and comprises the following steps: building a Kubernetes cluster system according to the predetermined number of Master servers and the number of Node child nodes; according to the target CPU architecture, exposing a physical network card bound on a Master server and a Node child node in advance to a Pod container for use by using an RDMA Shared mode; the method comprises the following steps: deploying a Multi CNI plug-in in a Kubernetes cluster system, and providing an additional network interface for a Pod container; the method comprises the following steps of: deploying a ring-pod plug-in used for RDMA (Remote Direct Memory Access) high-speed communication in a Master server and a Node child node of a Kubernetes cluster system; the IB communication function of the RDMA is started through additional network interfaces provided by a ring-pod plug-in and a Multi plug-in, and communication between pod containers is realized through a physical network card; according to the invention, multi-channel RDMA communication can be realized in the cluster, the limitation of the number of physical network cards is broken, and the cost is reduced.

Description

Technical Field

[0001] The present invention relates to a cross-node RDMA high-performance communication method in a Kubernetes cluster, and belongs to the technical field of computing networks. Background Art

[0002] With the development of cloud-native technology, the cost of using cloud for enterprises continues to increase. In order to improve resource utilization and reduce efficiency, the cloud cost system built with efficiency as the core will run through the entire cloud computing industry. As the de facto standard for container orchestration, Kubernetes can improve the availability of applications, shorten the deployment cycle, improve resource utilization, and provide a unified management platform for distributed systems. In a Kubernetes cluster, each node has multiple containers running. These containers may need to communicate with each other through the network, which is usually achieved through cluster-level network solutions, such as Flannel, Calico, etc., but these network plug-ins cannot meet high performance requirements. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method for cross-node RDMA high-performance communication in a Kubernetes cluster, which can realize multi-channel RDMA communication in the cluster, break the limitation of the number of physical network cards, and reduce costs.

[0004] To achieve the above object, the present invention is implemented by adopting the following technical solutions: In a first aspect, the present invention provides a method for cross-node RDMA high-performance communication in a Kubernetes cluster, comprising: Build a Kubernetes cluster system based on the predetermined number of Master servers and Node child nodes; According to the target CPU architecture, use RDMA Shared mode to expose the physical network cards pre-bound to the Master server and Node child nodes for use by the Pod container; Deploy the Multus CNI plug-in in the Kubernetes cluster system to provide additional network interfaces for the Pod container; Deploy the ring-pod plug-in for RDMA high-speed communication in the Master server and Node child nodes of the Kubernetes cluster system; Through the additional network interface provided by the ring-pod plug-in and the Multus plug-in, the IB communication function of RDMA is enabled, and communication between pod containers is achieved through the physical network card.

[0005] Furthermore, the method also includes: deploying the Canal network plug-in in the Kubernetes cluster system as the default network plug-in between Pod containers, implementing routing and access control of data between containers by modifying iptables and routing on each node, and coordinating node configuration information through etcd for basic communication between pods.

[0006] Furthermore, the access control part of the Canal network plug-in is implemented by Calico and configured in BGP mode, and the network part is implemented by Flannel, and the Flannel backend adopts Vxlan mode.

[0007] Furthermore, the server node includes one Master server node and at least one Node child node.

[0008] Furthermore, building the Kubernetes cluster system includes deploying docker service, kubectl software, kubelet software, kubeadm software, initializing the cluster environment, and adding Node child nodes to the Kubernetes cluster.

[0009] Furthermore, the RDMA Shared mode is used to expose the RoCE network card pre-bound to the Master server and the Node child node for use by the Pod container, including: Deploy the Shared Device Plugin to expose RDMA network card resources and schedule Pods, and customize the docker container image of the rdma-dev-plug plug-in to adapt to the domestic server architecture, ensure that the Kubernetes cluster can detect and identify the network card device, and set a certain number of available RDMA communication interfaces.

[0010] Furthermore, the docker container image of the custom rdma-dev-plug plug-in is adapted to the domestic server architecture, including: By customizing the docker container image of the rdma-dev-plug plug-in, setting export GO111MODUL=on, using go.mod to manage and install dependent packages, downloading dependencies to the vendor directory of the given path, porting supported modules to adapt to the domestic server architecture, building a new image, and configuring the configmap file according to the physical network card of the server, including custom resourceName, configuring the vendors and deviceIDs of the network card.

[0011] Furthermore, the Multus CNI plug-in is deployed in the Kubernetes cluster system to provide an additional network interface for the Pod container, including: By changing the configuration information in the yaml file, the default network of Kubernetes is changed to Canal, and the Multus CNI plug-in is deployed using the daemonset automatic installation method to provide additional network interfaces for the pods in the cluster.

[0012] Furthermore, the method also includes: designing a monitoring notification module in C / S mode, and utilizing the communication function between docker-client and docker-server to achieve memory resource isolation between multiple Pod containers, wherein the docker-client is located in the Pod container of RDMA communication, and the docker-server is located on the Master server node and the Node child node.

[0013] Furthermore, the communication function between docker-client and docker-server is used to implement memory resource isolation between multiple Pod containers, including: The dynamic link library libixml.so is managed during communication by setting the environment variable LD_PRELOAD. When a pod that needs RDMA submits a token application, docker_clinent registers the token in docker_server and queues the token. If there is no valid token, docker_server blocks the cuda program call, and RDMA communication cannot be performed until a valid token is obtained. If the pod container applies for a token, docker_clinent sends the actual communication information to docker_server through the protocol for real-time monitoring by docker_server. docker_server stores the relationship table between token information and pod information in the local memory of the node. The two tables uniquely identify a record through the pod_name field.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a method for cross-node RDMA high-performance communication in a Kubernetes cluster. Based on a domestic GPU architecture platform, in order to enable node communication in a Kubernetes cluster to meet the multi-channel RDMA high-performance communication requirements, the RDMA virtualization of virtual machines and containers is studied. The RDMA Shared mode is used to deploy the RDMA network card to the Kubernetes cluster using a hardware-assisted full virtualization technology. Multi-channel RDMA communication can be achieved in the cluster, breaking the limitation on the number of physical network cards, reducing costs, and avoiding performance interference by using a resource isolation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the internal network interface of the cluster of the present invention.

[0016] Figure 2 It is a flow chart of the present invention.

[0017] Figure 3 Flowchart of the pod resource isolation method. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0019] Embodiment 1: This embodiment introduces a method for cross-node RDMA high-performance communication in a Kubernetes cluster, including: Build a Kubernetes cluster system based on the predetermined number of Master servers and Node child nodes; According to the target CPU architecture, use RDMA Shared mode to expose the physical network cards pre-bound to the Master server and Node child nodes for use by the Pod container; Deploy the Multus CNI plug-in in the Kubernetes cluster system to provide additional network interfaces for the Pod container; Deploy the ring-pod plug-in for RDMA high-speed communication in the Master server and Node child nodes of the Kubernetes cluster system; Through the additional network interface provided by the ring-pod plug-in and the Multus plug-in, the IB communication function of RDMA is enabled, and communication between pod containers is achieved through the physical network card.

[0020] The method for cross-node RDMA high-performance communication in a Kubernetes cluster provided in this embodiment specifically involves the following steps: Step 1. First, complete the construction of the Kubernetes physical environment, where the server nodes include 1 Master server node and 1 Node child node. The Master node is the management center of the entire cluster, responsible for cluster management, container scheduling, state storage, etc. All components run on the Master node, and the Node node is the actual working node, responsible for running specific container deployment. The basic software dependencies required for the Kubernetes physical environment are docker service, kubectl software, kubelet software, kubeadm software, etc. Before deployment, name all nodes in the cluster system, then initialize the Kubernetes platform environment through the kubeadm init command, and finally add the Node child node to the Kubernetes cluster through the kubeadm join command.

[0021] Step 2. Plan and deploy network nodes in the Kubernetes cluster system. --network-plugin is set to CNI to enable and start the Kubernetes network service. Configure the Canal network plug-in based on the physical network card, use Canal as the default network plug-in between containers, and implement the network policy of Kubernetes. The access control part of Cannal is implemented by Calico, which is configured in BGP mode. The network part is implemented by Flannel, and the Fannel backend is in Vxlan mode. According to the characteristics of the domestic server architecture, it is necessary to customize the docker container image of the rdma-dev-plug plug-in. In the image, specify the domestic server as the compilation platform of docker, install the software environment of the domestic server, set exportGO111MODUL=on, use go.mod to manage and install the dependency package, and then download the dependency to the vendor directory of the given path, and transplant the supported modules to adapt to the domestic server architecture. After the image file is completed, use the command build to build the source code and build a new image. In the deployment of the rdma-dev-plug plug-in, you need to configure the configmap file according to the physical network card of the server, including: custom resourceName, configure the vendors and deviceIDs of the network card. Make sure that the Kubernetes cluster can detect and identify the network card device and set it to 500 available RDMA communication interfaces.

[0022] Step 3: Deploy the Multus CNI plug-in in the Kubernetes cluster system. According to the domestic server architecture, after the image file is produced, use the build command to compile the source code of the Multus CNI plug-in and build a new image. Change the cni-conf.jso configuration information in the yaml file, change the default network of Kubernetes to Canal, and use the daemonset automatic installation method to deploy the Multus CNI plug-in to provide additional network interfaces for the containers in the running cluster.

[0023] Step 4. Use the preset NetworkAttachmentDefinition custom network resource object to create Maclvan network configuration. Obtain the available Pod IP in the node through the Subnet file maintained by IPAM-Node, enable the Maclvan network card, and add the necessary IP configuration for the exposed available RDMA communication port. For example, you can deploy RDMA network communication pod1-pod10 on the Master server node, and deploy RDMA network communication pod1-pod10 on the Node node. Before the program runs, set the environment variable LD_PRELOAD to manage the dynamic link library libixml.so during communication. When the pod that needs RDMA submits a token application, docker_clinent will register this token with docker_server, queue the token, and ensure the priority of the task. If there is no valid token, docker_server will block the cuda program call, and RDMA communication will not be possible until a valid token is obtained. docker_clinent sends the memory actually used by this pod to docker_server through the protocol for real-time monitoring of docker_server. The relationship table between token information and pod information is stored in the local memory of the node. The two tables uniquely identify a record through the pod_name field. For example, when the docker_clinent of pod1-pod5 on the Master node has applied for a token from the docker_server and is performing RDMA transmission at the same time, the docker_server tracks pod1-pod5 through timestamps. At this time, pod6 also wants to perform RDMA transmission. The docker_server will query the total memory occupied by the five docker_clints. When the upper limit of the memory setting is exceeded, the docker_server will prohibit sending tokens to pod6. If it does not exceed the upper limit, the docker_server will send a token to pod6 and record it using the pod_name record and save it locally on the Master node.

[0024] Through this method, on the Kubernetes cluster of domestic servers, the Master node as the server and the Node node as the client can achieve multi-channel RDMA high-speed network communication, and use resource isolation methods to avoid performance interference.

[0025] The contents involved in the above embodiment are described below in conjunction with a preferred embodiment.

[0026] Before setting up the physical environment of the Kubernetes cluster, first determine that the server nodes include one Master server node and one Node child node, name the two nodes, set the hostname, and synchronize the execution time of all nodes. Install the software dependencies required for the Kubernetes physical environment, including docker service, kubectl software, kubelet software, kubeadm software, etc. Initialize the Kubernetes platform environment through the kubeadm init command, and add the Node child node to the Kubernetes cluster through the kubeadm join command.

[0027] Plan and deploy network nodes in the Kubernetes cluster system. Set --network-plugin to CNI to enable and start the Kubernetes network service. Configure the Canal network plug-in based on the physical network card, and use Canal as the main network plug-in between pods for basic communication between pods. The access control part of Cannal is implemented by Calico, which is configured in BGP mode. The network part is implemented by Flannel, and the Fannel backend is in Vxlan mode. After successful deployment, start a container, and there will be two virtual network interfaces loopback and eth0.

[0028] According to the characteristics of the domestic server architecture, customize the container image of the rdma-dev-plugin plug-in. Specify the domestic server as the compilation platform of docker in the dockerfile, install the software environment of the domestic server, set export GO111MODUL=on, use go.mod to manage and install the dependency packages, and then download the dependencies to the vendor directory of the given path, and transplant the supported modules to adapt to the domestic server architecture. After the image file is produced, use the build command to compile the rdma-dev-plugin source code and build the image. In the deployment of the rdma-dev-pluin plug-in, it is necessary to configure the configmap file according to the physical network card of the Master server and the Node child node, including: custom resourceName, and configure the vendors and deviceIDs of the network card according to the manufacturer of the network card. As follows: "resourceName": "hca_shared_devices_b", "rdmaHcaMax": 500, "selectors": { "vendors": ["15b3"], "deviceIDs": ["1013"], "ifNames": ["ib3", "ib4"] } Ensure that all nodes in the Kubernetes cluster can detect and identify their respective network card devices and set them to 500 allocatable virtualized RDMA communication interfaces.

[0029] Deploy the Multus CNI plug-in in the Kubernetes cluster system. According to the domestic server architecture, after the dockerfile file is created, use the build command to compile the Multus CNI source code and build the image. Change the cni-conf.jso configuration information in the yaml file, change the default network of Kubernetes to Canal, and use the daemonset automatic installation method to deploy the Multus CNI plug-in to provide additional network interfaces for the pods in the running cluster. After the Multus CNI plug-in is deployed, a Multus binary executable file is generated at / opt / cni / bin / on each server node to configure the pod's network.

[0030] Use the preset custom network resource object (NetworkAttachmentDefinition) to create the Maclvan network configuration. Obtain the available Pod IP in the node through the Subnet file maintained by IPAM-Node, enable the Maclvan network card, and add the necessary IP configuration for the exposed available RDMA communication interface. Install and deploy RDMA pod1 on the Master server node, and install and deploy RDMA test pod2 on the Node child node. On the two domestic server platforms, enter the communication container and enter the pod namespace to view the IP. Net1 is the IP address obtained using the Canal network.

[0031] Deploy RDMA network communication pod1-pod10 on the Master server node, and deploy RDMA network communication pod1-pod10 on the Node node. Before the program runs, set the environment variable LD_PRELOAD to manage the dynamic link library libixml.so during communication. When the pod that needs RDMA submits a token application, docker_clinent will register the token in docker_server, queue the token, and ensure the priority of the task. If there is no valid token, docker_server will block the cuda program call, and RDMA communication will not be possible until a valid token is obtained. docker_clinent sends the actual memory used by this pod to docker_server through the protocol for real-time monitoring of docker_server. The relationship table between token information and pod information is stored in the local memory of the node. The two tables uniquely identify a record through the pod_name field. The contents of the token information table and the pod information table are shown in the following table. When the Pod task ends, the token will expire, and the docker_server will revoke the pod token and select the next token from the token queue to issue it to the corresponding pod.

[0032] Table 1 pod information table

[0033] Table 2 Token information table

[0034] For example, when the docker_clinents of pod1-pod5 on the Master node have applied for tokens from the docker_server and are performing RDMA transmission at the same time, the docker_server tracks pod1-pod5 through timestamps. At this time, pod6 also wants to perform RDMA transmission, and the docker_server will query the total memory occupied by the five docker_clients. If it exceeds the upper limit of the memory setting, the docker_server will prohibit sending tokens to pod6. If it does not exceed the upper limit, the docker_server will send a token to pod6 and record it using the pod_name record and save it locally on the Master node.

[0035] The present invention is based on hardware-assisted RDMA virtualization communication technology, configures the hardware network card in the cluster, and uses Maclavn network direct mode combined with RDMA's IB communication technology to achieve multi-channel RDMA high-performance communication across nodes in the cluster. In this cluster, the number of pods that can be used for RDMA communication can reach up to 500. When pod resources are over-utilized, it is easy to cause RDMA performance interference. Therefore, a token-timed C / S mode monitoring notification module is designed to achieve pod resource isolation. This module also has its own low extra overhead characteristics. This method can not only solve the problem of slow high-density data transmission rate and low throughput between nodes in the cluster, but also meet the deployment requirements of the cloud environment, with the advantages of high bandwidth, low latency, isolatable, portable and controllable.

[0036] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for high-performance cross-node RDMA communication in a Kubernetes cluster, characterized in that: include: Build a Kubernetes cluster system based on the predetermined number of Master servers and Node child nodes; According to the target CPU architecture, use RDMA Shared mode to expose the physical network cards pre-bound to the Master server and Node child nodes for use by the Pod container; Deploy the Multus CNI plug-in in the Kubernetes cluster system to provide additional network interfaces for the Pod container; Deploy the ring-pod plug-in for RDMA high-speed communication in the Master server and Node child nodes of the Kubernetes cluster system; Through the additional network interface provided by the ring-pod plug-in and the Multus plug-in, the IB communication function of RDMA is enabled, and communication between pod containers is achieved through the physical network card.

2. The method for cross-node RDMA high-performance communication in a Kubernetes cluster according to claim 1, characterized in that: The method also includes: deploying the Canal network plug-in in the Kubernetes cluster system as the default network plug-in between Pod containers, implementing routing and access control of data between containers by modifying iptables and routing on each node, and coordinating node configuration information through etcd for basic communication between pods.

3. The method for cross-node RDMA high-performance communication in a Kubernetes cluster according to claim 2, characterized in that: The access control part of the Canal network plug-in is implemented by Calico and configured in BGP mode, and the network part is implemented by Flannel, and the Flannel backend adopts Vxlan mode.

4. The method for cross-node RDMA high-performance communication in a Kubernetes cluster according to claim 1, characterized in that: The server nodes include one Master server node and at least one Node child node.

5. The method for cross-node RDMA high-performance communication in a Kubernetes cluster according to claim 1, characterized in that: The Kubernetes cluster system is built by deploying docker service, kubectl software, kubelet software, kubeadm software, initializing the cluster environment, and adding Node child nodes to the Kubernetes cluster.

6. The method for cross-node RDMA high-performance communication in a Kubernetes cluster according to claim 1, characterized in that: The RDMA Shared mode is used to expose the RoCE network card pre-bound to the Master server and Node child nodes for use by the Pod container, including: Deploy the Shared Device Plugin to expose RDMA network card resources and schedule Pods, and customize the docker container image of the rdma-dev-plug plug-in to adapt to the domestic server architecture, ensure that the Kubernetes cluster can detect and identify the network card device, and set a certain number of available RDMA communication interfaces.

7. The method for cross-node RDMA high-performance communication in a Kubernetes cluster according to claim 6, characterized in that: The custom rdma-dev-plug plug-in docker container image is adapted to the domestic server architecture, including: By customizing the docker container image of the rdma-dev-plug plug-in, setting export GO111MODUL=on, using go.mod to manage and install dependent packages, downloading dependencies to the vendor directory of the given path, porting supported modules to adapt to the domestic server architecture, building a new image, and configuring the configmap file according to the physical network card of the server, including custom resourceName, configuring the vendors and deviceIDs of the network card.

8. The method for cross-node RDMA high-performance communication in a Kubernetes cluster according to claim 1, characterized in that: The Multus CNI plug-in is deployed in the Kubernetes cluster system to provide additional network interfaces for the Pod container, including: By changing the configuration information in the yaml file, the default network of Kubernetes is changed to Canal, and the Multus CNI plug-in is deployed using the daemonset automatic installation method to provide additional network interfaces for the pods in the cluster.

9. The method for cross-node RDMA high-performance communication in a Kubernetes cluster according to claim 1, characterized in that: The method also includes: designing a monitoring notification module in C / S mode, and utilizing the communication function between docker-client and docker-server to implement memory resource isolation between multiple Pod containers, wherein the docker-client is located in the Pod container of RDMA communication, and the docker-server is located on the Master server node and the Node child node.

10. The method for cross-node RDMA high-performance communication in a Kubernetes cluster according to claim 9, characterized in that: The communication function between docker-client and docker-server is used to implement memory resource isolation between multiple Pod containers, including: The dynamic link library libixml.so is managed during communication by setting the environment variable LD_PRELOAD. When a pod that needs RDMA submits a token application, docker_clinent registers the token in docker_server and queues the token. If there is no valid token, docker_server blocks the cuda program call, and RDMA communication cannot be performed until a valid token is obtained. If the pod container applies for a token, docker_clinent sends the actual communication information to docker_server through the protocol for real-time monitoring by docker_server. docker_server stores the relationship table between token information and pod information in the local memory of the node. The two tables uniquely identify a record through the pod_name field.

Images