A k8s cluster network quality detection method

Abstract

The application discloses a k8s cluster network quality detection method, first, a message passing interface (MPI) container image is constructed; then, the MPI container image is used to create a plurality of Pods in the k8s cluster, and an MPI cluster is established through the Pods; next, an MPI parallel program for calculating the delay between the Pods is developed; an MPI parallel program for calculating the bandwidth between the MPI Pods is developed; finally, data is collected and network quality conditions are analyzed. The application avoids the cumbersome process of manual installation, configuration and scheduling in the prior art, and greatly reduces operation and maintenance costs and manual operation risks.

Description

Technical Field

[0001] This invention belongs to the field of network technology, specifically relating to a method for detecting network quality in a Kubernetes cluster. Background Technology

[0002] Kubernetes (k8s) is an open-source distributed container orchestration platform that automates the deployment, management, and scaling of containerized applications. It is currently widely used in cloud computing, big data, microservices, and other scenarios.

[0003] Kubernetes networks are characterized by high flexibility, strong scalability, and high reliability, but they also have disadvantages such as complex structure and steep learning curve. Especially as a distributed cluster, network construction is very complex, and problems can easily occur at both the physical and software levels. Therefore, it is very important to quickly and easily obtain the network quality status of the distributed network within the cluster. This includes the network quality status between cluster nodes and between Pods, and the network quality status mainly includes network connectivity, network latency, and network bandwidth.

[0004] Existing Kubernetes (Kubernetes) operation and maintenance tools and monitoring methods primarily focus on monitoring the utilization of node resources such as CPU, memory, and network interface cards (NICs). There are many monitoring tools available, and operation and maintenance tools and monitoring methods mainly include Prometheus + Grafana, Kubernetes Metrics Server, and ELK Stack. However, they lack the ability to automate and fine-grained measure and diagnose inter-node network performance (especially latency and bandwidth). The implementation process typically includes the following steps: Monitoring data collection: Deploy monitoring agents such as Metrics Server or Node Exporter within each Kubernetes node or Pod to periodically collect resource utilization metrics such as node CPU, memory, disk, and network interface traffic.

[0005] Data transmission and storage: Monitoring data is reported to the monitoring system (such as Prometheus) via the Kubernetes API Server or directly via an HTTP interface, where it is stored in a time-series format.

[0006] Data query and visualization: Users can query metric data through Prometheus or Grafana and use predefined dashboards to display curves showing changes in metrics such as CPU utilization, memory usage, and total network I / O.

[0007] Alarm mechanism: When a metric exceeds a threshold (such as CPU utilization > 90%), an alarm is triggered, and the administrator is notified via email, WebHook, or other means.

[0008] Existing Kubernetes (k8s) operation and maintenance tools have the following shortcomings in monitoring node network performance: Lack of direct measurement of inter-node network latency and bandwidth: Existing tools can only collect basic indicators such as network card throughput and packet loss rate, and cannot perform real-time, automated testing of point-to-point network performance between nodes.

[0009] It cannot reflect the true communication quality of the network link: Normal network card traffic on a single node does not necessarily mean low latency or sufficient bandwidth in inter-node communication. Existing tools cannot pinpoint performance bottlenecks in cross-node links.

[0010] The testing process is not automated and lacks dynamic adaptation capabilities. The existing monitoring only collects metrics. To test the network performance between nodes, the administrator needs to manually log in to the node and manually run network testing tools (such as iperf), which cannot be dynamically expanded with the cluster size. Summary of the Invention

[0011] To overcome the shortcomings of existing technologies, this invention provides a method for network quality detection in a Kubernetes cluster. First, a Message Passing Interface (MPI) container image is constructed. Then, multiple Pods are created using the MPI container image in the Kubernetes cluster, and an MPI cluster is established using these Pods. Next, a parallel MPI program is developed to calculate the latency between Pods; a parallel MPI program is also developed to calculate the bandwidth between MPI Pods; finally, data is collected and the network quality is analyzed. This invention avoids the cumbersome process of manual installation, configuration, and scheduling in existing technologies, significantly reducing operational costs and the risks associated with manual operations.

[0012] The technical solution adopted by this invention to solve its technical problem is as follows: Step 1: Build the MPI container image; Step 2: Use Pods to build an MPI cluster in the Kubernetes cluster; a Pod is the smallest deployable computing unit that can be created and managed in Kubernetes. Step 3: Develop an MPI parallel program to calculate the latency between MPI Pods; Step 4: Develop an MPI parallel program to calculate the bandwidth between MPI Pods; Step 5: Collect data and analyze network quality.

[0013] Preferably, step 1 specifically comprises: Step 1-1: The MPI container image contains an MPI environment and OpenSSH server software, and generates a pair of OpenSSH key files in the .ssh directory under the user's directory, where the private key file is id_rsa and the public key file is id_rsa.pub. Steps 1-2: Preparation for passwordless SSH (Secure Shell) login between MPI Pods: Create an authorized_keys file in the .ssh directory of your user directory and append the contents of the public key id_rsa.pub file to the authorized_keys file.

[0014] Preferably, step 2 specifically comprises: Step 2-1: Create a Kubernetes statefulset workload using an MPI container image. This workload is based on the statefulset controller and uses the Kubernetes affinity mechanism Pod Anti-Affinity configuration to schedule each MPI Pod within the workload to different nodes, thereby ensuring that only one MPI Pod is deployed on each node, thus completing the automated deployment of the MPI cluster. Step 2-2: Ensure that the openssh server software is running in each Pod; each MPI Pod is equivalent to an MPI node, and eventually all MPI Pods form an MPI cluster; Steps 2-3: Collect the IP of each MPI Pod, log in to the MPI Pod on the Kubernetes master node, create the hostfile file required for the MPI cluster, and add the IP of all MPI Pods in the cluster and the corresponding number of CPU cores to the hostfile file; Steps 2-4: Leveraging the resource characteristics of Kubernetes, each MPI Pod resides on a separate Kubernetes node, and communication between Pods is accomplished based on the Kubernetes Pod network. Preferably, the k8s Pod network is a virtualized network built on the communication network between k8s cluster nodes and depends on the communication network between k8s cluster nodes.

[0015] Preferably, step 3 specifically comprises: In the MPI parallel program, each MPI Pod tests the latency of other MPI Pods sequentially. Then, using the parallel execution characteristics and point-to-point communication mechanism of MPI, the latency between any two MPI Pods is calculated. The latency calculation between MPI Pods within the cluster is automatically executed by the MPI program and the data is automatically saved. Since the MPI Pods will affect each other when performing these calculations, only one pair of MPI Pods' latency is calculated at a time.

[0016] Preferably, the delay calculation uses the Ping-Pong method, where one node sends a message to another node, and the other node replies after receiving the message. In this case, the first node calculates the time spent on one message communication, which is twice the delay between the two nodes.

[0017] Preferably, step 4 specifically comprises: In the MPI parallel program, each MPI Pod tests the bandwidth of other MPI Pods in turn. Then, using the parallel execution feature and point-to-point communication mechanism of MPI, the bandwidth of any two MPI Pods is calculated. The bandwidth calculation between MPI Pods within the cluster is automatically executed by the MPI program and the data is automatically saved. Since the MPI Pods will affect each other when performing these calculations, only one pair of MPI Pods is calculated at a time.

[0018] Preferably, the bandwidth is calculated by sending a certain amount of data from one node to another node for a period of time, the other node receiving the data, and finally dividing the total amount of data sent by the duration to obtain the communication bandwidth.

[0019] Preferably, step 5 specifically comprises: If the latency between two MPI Pods times out and they cannot connect, then there is a network failure between the two MPI Pods or the Kubernetes node they reside on. If the latency between two MPI Pods exceeds a set threshold, the network communication between these two MPI Pods or the Kubernetes node they reside on is poor and cannot meet the cluster's usage requirements.

[0020] The beneficial effects of this invention are as follows: 1. Automated deployment and testing; By combining pre-built MPI container images with Kubernetes' StatefulSet controller and scheduling policies, MPI clusters can be automatically deployed in Kubernetes clusters, avoiding the tedious process of manual installation, configuration and scheduling in existing technologies, and significantly reducing operation and maintenance costs and the risks of manual operation.

[0021] 2. Highly efficient parallel testing and data collection; By leveraging the parallel computing capabilities of MPI, the measurement speed of inter-node latency and bandwidth can be accelerated while ensuring test accuracy. Furthermore, through automated data collection and storage, the workload of manual data processing and statistics can be reduced, resulting in a significant improvement in overall test efficiency.

[0022] 3. Rapid fault location and visual display; Based on the collection of latency and bandwidth data, this application can automatically generate charts and support threshold judgment, quickly identify network bottlenecks or single points of failure, provide a clear basis for subsequent fault diagnosis, and reduce fault investigation time.

[0023] 4. Improve operational efficiency and system stability; Compared to traditional manual testing methods, this application improves automation, real-time performance, and diagnostic accuracy, reducing the impact of network failures on business systems and ensuring the stable operation of Kubernetes clusters in large-scale distributed environments. Attached Figure Description

[0024] Figure 1 This is a diagram of the method architecture of the present invention; Figure 2 This is a flowchart of Embodiment 1 of the present invention; Figure 3 This is a flowchart of Embodiment 2 of the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] The purpose of this invention is as follows: 1. Build a network performance testing system that can be automatically deployed and dynamically adapted within a Kubernetes cluster; 2. Enable real-time, automated measurement of inter-node latency and bandwidth; 3. By using distributed testing and visualization analysis, network bottlenecks and abnormal nodes can be quickly located, thereby overcoming the shortcomings of existing technologies in terms of monitoring dimensions, automation level and fault diagnosis efficiency, and meeting the needs of large-scale Kubernetes clusters for refined network performance monitoring and intelligent operation and maintenance.

[0027] This invention proposes a network quality monitoring method applicable to any normally operating Kubernetes cluster. It aims to address the lack of efficient and automated network latency and bandwidth monitoring methods in existing Kubernetes clusters, facilitating rapid identification of network performance bottlenecks and fault ranges. The method mainly includes the following components, and its architecture diagram is shown below. Figure 1 As shown.

[0028] 1. Prepare the MPI container image; First, prepare and build an MPI container image, which is required to establish an MPI cluster. The container image installs the MPI environment and OpenSSH server software, and generates a pair of OpenSSH key files in the `.ssh` directory under the user's home directory. The private key file is `id_rsa`, and the public key file is `id_rsa.pub`. To prepare for passwordless SSH login between MPI Pods, create an `authorized_keys` file in the `.ssh` directory under the user's home directory and append the contents of the `id_rsa.pub` public key file to the `authorized_keys` file.

[0029] 2. Design an MPI cluster using Pods within a Kubernetes cluster; An MPI container image is used to create a Kubernetes statefulset workload. This workload is based on the Kubernetes statefulset controller and, combined with Kubernetes' Pod Anti-Affinity configuration, schedules each MPI Pod within the workload to different nodes. This ensures that only one MPI Pod is deployed on each node, thus automating the deployment of the MPI cluster. Each Pod is guaranteed to have the OpenSSH server software running. Each MPI Pod is equivalent to an MPI node, and ultimately, all MPI Pods form an MPI cluster.

[0030] Collect the IP address of each MPI Pod, log in to the MPI Pod on the Kubernetes master node, create the hostfile file required for the MPI cluster, and add the IP addresses and corresponding CPU core counts of all MPI Pods in the cluster to the hostfile file.

[0031] By leveraging the resource characteristics of Kubernetes (k8s), each MPI Pod resides on a separate k8s node and communication between Pods is achieved through a k8s Pod network. This k8s Pod network is a virtualized network built upon the communication network between k8s cluster nodes and relies on it. Therefore, network communication between all MPI Pods reflects not only the network quality between Pods but also the network quality between k8s cluster hosts. This automated distributed deployment method provides node-level independent observation points for network monitoring, unlike existing manual deployment and centralized testing solutions.

[0032] 3. Develop an MPI parallel program to calculate the latency between MPI Pods; In MPI, latency calculation typically uses the Ping-Pong method, where one node sends a message to another, and the other node responds upon receiving the message. The first node then calculates the time spent on this single message communication, which is twice the latency between the two nodes. In the parallel MPI program developed here, any MPI Pod sequentially tests the latency of other MPI Pods. Leveraging MPI's parallel execution capabilities and point-to-point communication mechanism, the latency between any two MPI Pods can be calculated. Latency calculations between MPI Pods within the cluster are automatically executed by the MPI program, and the data is automatically saved. Since MPI Pods may interfere with each other during these calculations, latency calculations can be performed on only one pair of MPI Pods at a time. The above test program supports parameterized configuration, such as packet size, test duration, number of retries, and timeout. These parameters can be flexibly set according to cluster size, network conditions, and test requirements to ensure stable and reproducible test results.

[0033] 4. Develop an MPI parallel program to calculate the bandwidth between MPI Pods; The bandwidth calculation method here involves sending a certain amount of data from one node to another for a specified period of time, with the other node receiving the data. The communication bandwidth is then calculated by dividing the total amount of data sent by the duration. In the MPI parallel program developed here, any MPI Pod sequentially tests the bandwidth of other MPI Pods. Leveraging the parallel execution characteristics and point-to-point communication mechanism of MPI, the bandwidth of any two MPI Pods can be calculated. Furthermore, the bandwidth calculations between MPI Pods within the cluster are automatically executed by the MPI program, and the data is automatically saved. Since MPI Pods may interfere with each other during these calculations, only one pair of MPI Pods is allowed to perform bandwidth calculations at a time. The above test program also supports parameterized configuration, such as packet size, test duration, number of retries, and timeout. These parameters can be flexibly set according to cluster size, network conditions, and test requirements to ensure stable and reproducible test results.

[0034] 5. Collect data and analyze network quality; The above procedure allows for the rapid acquisition of latency and bandwidth data between any two MPI Pods in a Kubernetes cluster. Since these MPI Pods can utilize nearly all network resources of their respective nodes, the data reflects the network quality between the Kubernetes nodes. The data is then sorted and visualized using bar charts, line graphs, and other graphs. If the latency between two MPI Pods times out and they cannot connect, there is a network connectivity issue with either the two MPI Pods or their respective Kubernetes nodes. High latency between two MPI Pods indicates poor network communication between them or their respective Kubernetes nodes, failing to meet cluster usage requirements. There may also be cases where the bandwidth between two MPI Pods does not meet expectations. This network quality data can be collected conveniently and quickly through the above process and steps, and combined with latency / bandwidth threshold determination, it supports the rapid location of single points of failure and global performance issues.

[0035] In summary, the method of this invention not only realizes the automated measurement and analysis of network latency and bandwidth between Kubernetes cluster nodes, but also enables rapid location of fault range when performance is abnormal, solving the problems of low network monitoring efficiency and inaccurate fault location in the prior art.

[0036] Example 1: The flowchart of Example 1 is as follows Figure 2 As shown.

[0037] This embodiment provides a method for implementing network latency collection in a Kubernetes cluster, supporting both fast collection mode and non-concurrent collection mode. The specific steps are as follows: 1. System environment and preparation; This embodiment runs on any Kubernetes cluster, taking Kubernetes version v1.24+ as an example. The cluster contains 10 worker nodes, each configured with 8 CPU cores and 8GB of memory, and uses Flannel or Calico network plugins as CNI plugins. The MPI container image is built on Ubuntu 20.04, with OpenMPI 4.1.5 and OpenSSH 8.9p1 installed. The following command is executed within the image to generate a key pair, and the public key is written to the authorized_keys file to ensure passwordless communication between MPI nodes: ssh-keygen -t rsa -b 2048 -f / root / .ssh / id_rsa -N "" cat / root / .ssh / id_rsa.pub>> / root / .ssh / authorized_keys 2. Automated deployment of MPI clusters; Create 10 MPI Pods (the same number of nodes) using the StatefulSet controller. Configure the PodAntiAffinity policy to ensure that each MPI Pod is scheduled to a different physical node. Collect the IP addresses of all MPI Pods and generate the / root / hostfile file. An example file content is shown below: 10.0.0.1 slots=4 10.0.0.2 slots=4 ... 10.0.0.10 slots=4 Where slots represents the number of CPU cores in a node.

[0038] 3. Collect network latency data; Start and execute the Ping-Pong test program on the MPI master node to test the latency between any two MPI Pods: Node A sends a message of length L to node B, and node B immediately returns a response. Node A records the message round-trip time T and calculates the one-way delay:

[0039] L is set to 8 bytes, each pair of nodes is tested 10 times, and the average value is taken as the final result.

[0040] During the test, the program only allows a pair of nodes to be tested for latency at the same time to eliminate interference.

[0041] 4. Data collection and analysis; Test results are automatically written to a CSV file, and bar charts, line graphs, and other graphs are generated using tools such as Matplotlib. If the delay exceeds a specified time, such as 50ms, or if a connection times out, a network error is flagged.

[0042] Through the above steps, this embodiment can quickly and accurately collect network latency data between all nodes in a Kubernetes cluster, supports different collection modes, and facilitates subsequent network diagnosis and optimization.

[0043] Example 2: The flowchart of Example 2 is as follows Figure 3 As shown.

[0044] This embodiment provides a method for measuring the network bandwidth between MPI Pods in a Kubernetes cluster. The specific steps are as follows: 1. System environment and preparation; Similar to Example 1, the same Kubernetes cluster containing 10 worker nodes can be used.

[0045] 2. Automated deployment of MPI clusters; As in Example 1, the same MPI container image can be used.

[0046] 3. Collect network bandwidth; Node A sends a total of D data to node B within a duration T. Node B records the amount of data received and calculates the bandwidth.

[0047] Where D=100MB, T=10 seconds, and the data block size is 64KB.

[0048] To ensure testing accuracy, the bandwidth testing program is designed to run only one pair of nodes at any given time, avoiding bandwidth contention caused by multiple pairs of nodes sending data simultaneously.

[0049] 4. Data collection and analysis; All test results are written to the database, automatically generating bar charts, line graphs, and other graphs. If a node's bandwidth is lower than the expected value, such as 1Gbps, it is marked as an abnormal connection.

[0050] This embodiment can visually demonstrate the bandwidth distribution among nodes in the cluster, making it easier to identify potential low-bandwidth bottleneck nodes.

Claims

1. A method for detecting the quality of a Kubernetes cluster network, characterized in that, Includes the following steps: Step 1: Build the MPI container image; Step 2: Use Pods to build an MPI cluster in the Kubernetes cluster; a Pod is the smallest deployable computing unit that can be created and managed in Kubernetes. Step 3: Develop an MPI parallel program to calculate the latency between MPI Pods; Step 4: Develop an MPI parallel program to calculate the bandwidth between MPI Pods; Step 5: Collect data and analyze network quality.

2. The method for detecting the quality of a Kubernetes cluster network according to claim 1, characterized in that, Step 1 specifically involves: Step 1-1: The MPI container image contains an MPI environment and OpenSSH server software, and generates a pair of OpenSSH key files in the .ssh directory under the user's directory, where the private key file is id_rsa and the public key file is id_rsa.pub. Steps 1-2: Preparation for passwordless SSH login between MPI Pods: Create an authorized_keys file in the .ssh directory of your user directory and append the contents of the public key id_rsa.pub file to the authorized_keys file.

3. The method for detecting the quality of a Kubernetes cluster network according to claim 2, characterized in that, Step 2 specifically involves: Step 2-1: Create a Kubernetes statefulset workload using an MPI container image. This workload is based on the statefulset controller and uses the Kubernetes affinity mechanism Pod Anti-Affinity configuration to schedule each MPI Pod within the workload to different nodes, thereby ensuring that only one MPI Pod is deployed on each node, thus completing the automated deployment of the MPI cluster. Step 2-2: Ensure that the openssh server software is running in each Pod; each MPI Pod is equivalent to an MPI node, and eventually all MPI Pods form an MPI cluster; Steps 2-3: Collect the IP of each MPI Pod, log in to the MPI Pod on the Kubernetes master node, create the hostfile file required for the MPI cluster, and add the IP of all MPI Pods in the cluster and the corresponding number of CPU cores to the hostfile file; Steps 2-4: Leveraging the resource characteristics of Kubernetes, each MPI Pod resides on a separate Kubernetes node, and communication between Pods is accomplished based on the Kubernetes Pod network.

4. The method for detecting the quality of a Kubernetes cluster network according to claim 3, characterized in that, The k8s Pod network is a virtualized network built on the communication network between k8s cluster nodes and depends on the communication network between k8s cluster nodes.

5. The method for detecting the quality of a Kubernetes cluster network according to claim 4, characterized in that, Step 3 specifically involves: In the MPI parallel program, each MPI Pod tests the latency of other MPI Pods sequentially. Then, using the parallel execution characteristics and point-to-point communication mechanism of MPI, the latency between any two MPI Pods is calculated. The latency calculation between MPI Pods within the cluster is automatically executed by the MPI program and the data is automatically saved. Since the MPI Pods will affect each other when performing these calculations, only one pair of MPI Pods' latency is calculated at a time.

6. The method for detecting the quality of a Kubernetes cluster network according to claim 5, characterized in that, The delay calculation is based on the Ping-Pong method, where one node sends a message to another node, and the other node responds after receiving the message. The first node then calculates the time spent on one message communication, which is twice the delay between the two nodes.

7. The method for detecting the quality of a Kubernetes cluster network according to claim 6, characterized in that, Step 4 specifically involves: In the MPI parallel program, each MPI Pod tests the bandwidth of other MPI Pods in turn. Then, using the parallel execution feature and point-to-point communication mechanism of MPI, the bandwidth of any two MPI Pods is calculated. The bandwidth calculation between MPI Pods within the cluster is automatically executed by the MPI program and the data is automatically saved. Since the MPI Pods will affect each other when performing these calculations, only one pair of MPI Pods is calculated at a time.

8. The method for detecting the quality of a Kubernetes cluster network according to claim 7, characterized in that, The bandwidth is calculated by sending a certain amount of data from one node to another node for a period of time, receiving the data at the other node, and finally dividing the total amount of data sent by the duration to obtain the communication bandwidth.

9. The method for detecting the quality of a Kubernetes cluster network according to claim 8, characterized in that, Step 5 specifically involves: If the latency between two MPI Pods times out and they cannot connect, then there is a network failure between the two MPI Pods or the Kubernetes node they reside on. If the latency between two MPI Pods exceeds a set threshold, the network communication between these two MPI Pods or the Kubernetes node they reside on is poor and cannot meet the cluster's usage requirements.

Images