A Kubernetes-based distributed pod scheduling optimization method combined with Flink

Abstract

The application discloses a kind of distributed Pod scheduling optimization methods based on Kubernetes combination Flink, including the following steps: step one, create a central cluster based on Kubernetes;Step two, when preparing to set Pod under a Node node in cluster as exposed to outside, resource early warning model is constructed based on Markov chain;Step three, the data between the exposed Pod of multiple clusters is mapped as TaskManager, using resource adaptation model combined with Task Slot, SubTask after optimization by Pareto optimal solution algorithm is sequentially placed into different Task Slot, complete the uniform distribution of SubTask on TaskManager.This method uses Markov chain to construct resource early warning model to solve the problem of fault prediction before Pod exposure port, reduces the situation that Master allocation scheduling resource cannot predict Pod running health, uses Pareto classification method to construct resource adaptation model to solve the memory slot problem of Flink subtask allocation, reduces the overhead of task, reduces the data handover and cache between threads, and improves the overall throughput, reduces the processing delay.

Description

Technical Field

[0001] This invention belongs to the field of scheduling optimization technology, specifically relating to a distributed Pod scheduling optimization method based on Kubernetes and Flink. Background Technology

[0002] As digitalization deepens, the number of operational devices in various organizations is gradually increasing. In data center operations and maintenance scenarios, cluster business scale is large, application relationships are complex, and dependency layers are multiple, making troubleshooting difficult. Even though operations and maintenance are evolving from manual to tool-based and platform-based methods, they still cannot meet the current cluster's requirements for operations and maintenance management and efficient scheduling. Existing technologies struggle to automatically predict port failures exposed by Pods in a cluster when they interact with other clusters, thus increasing the probability of data exchange anomalies between distributed clusters and increasing the uncertainty of business interactions between Pods in distributed clusters.

[0003] Chinese invention patent CN202010653672.8 discloses a cloud resource scheduling method and system based on Kubernetes. The method includes: creating a Pod based on a resource object creation request, recording the required scheduler type in the resource object creation request to the configuration file of the created Pod; after detecting the created Pod, selecting a scheduler from a pre-deployed scheduler group of the required scheduler type, and updating the identifier of the selected scheduler to the Pod's configuration file; the scheduler group contains multiple schedulers of the same type; after the selected scheduler detects its own identifier, it determines the cluster node to be assigned to the Pod and sends a binding request; after receiving the binding request through the admission controller, it verifies that the resources still meet the requirements of the required resources and records the identifier of the cluster node to the Pod; after the corresponding cluster node detects that the Pod has recorded the cluster node's identifier, it starts the Pod and creates the corresponding container. However, this scheduling method cannot perform ranking and analysis of Pod scheduling priorities and trends.

[0004] Chinese invention patent CN202111534118.9 discloses a container resource scheduling method, apparatus, computer device, and storage medium. This method constructs a Markov chain directed graph by forming multiple state nodes from various runtime resources within a container. It obtains the transition probability matrix corresponding to the Markov chain directed graph based on the historical state transition counts between the multiple state nodes. Based on the transition probability matrix, the number of runtime resources in the container, and the dominant runtime resource used by the service currently using the most runtime resources in the container, a second dominant runtime resource is determined for the container. The runtime resources in the container are then scheduled according to this second dominant runtime resource. However, this resource scheduling method cannot adapt resources for subtasks involving data computation mapping between Pods. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by providing a distributed Pod scheduling optimization method based on Kubernetes and Flink. The invention primarily employs a Markov chain to construct a resource early warning model to solve the problem of fault prediction before Pods expose ports, reducing the likelihood that the Master will allocate scheduling resources based solely on resource optimization without predicting Pod health. It also uses the Pareto classification method to construct a resource adaptation model to solve the problem of subtask memory slot allocation in Flink, reducing task overhead, minimizing data handover and caching between threads, and improving overall throughput while reducing processing latency.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A distributed Pod scheduling optimization method based on Kubernetes and Flink includes the following steps:

[0008] Step 1: Create a central Kubernetes-based cluster on the cluster management server;

[0009] Step 2: When preparing to set a Pod under a Node in the cluster to be exposed to the outside world, build a resource early warning model based on Markov chains;

[0010] Step 3: Data transmission between exposed Pods in multiple clusters is mapped to TaskManager. Using the resource adaptation model combined with Flink's Task Slot, SubTasks optimized by the Pareto optimal solution algorithm are sequentially placed into different Task Slots to achieve a uniform distribution of SubTasks on the TaskManager.

[0011] Kubernetes is an open-source Linux container automation platform used to manage multiple hosts forming a cluster. The TaskManager executes data stream tasks or subtasks, caching and interacting with data streams. At least one TaskManager is required. The Task Slot is a process slot, with a default number equal to the number of CPU cores. Subtasks are evenly distributed across TaskManagers with multiple Task Slots, sharing a single process, TCP connections, and heartbeat messages. They can also share datasets and data structures, reducing task overhead. Furthermore, chaining operators with overlapping operations across multiple Pods into the same Subtask reduces data handover and caching between threads, improves overall throughput, and reduces processing latency.

[0012] Further explanation of the present invention: the Kubernetes-based cluster mainly includes Master, Node, and Pod, and its analysis method includes:

[0013] Step 1: Deploy the Master on the cluster management server and store the configuration files for the central and local Pods of the distributed cluster.

[0014] Step 2: Obtain historical image data from the image database, including server configuration data for each version of the Pod, and sort the values ​​from largest to smallest to determine the Pod scheduling priority.

[0015] The Master is the primary node, also known as the cluster control node, which is responsible for the management and control of the entire cluster; the Node is a node, also known as a workload node, which is used to execute the workloads assigned by the Master; the Pod is a collection of one or more containers that are deployed as a whole to a single node.

[0016] The server configuration data for each version of the Pod includes CPU utilization, memory usage, disk I / O, and network I / O.

[0017] A further explanation of the present invention is that the resource early warning model formula is:

[0018]

[0019] In the formula: X(k) represents the state vector of the trend analysis and prediction object at time t=k, P represents the one-step transition probability matrix, and X(k+1) represents the state vector of the trend analysis and prediction object at time t=k+1.

[0020] The resource early warning model is used to solve the problem of fault prediction before a Pod exposes its port, rather than using the Pod with the most idle resources as the exposed port.

[0021] A further explanation of this invention is that the resource adaptation model is constructed using the Pareto analysis method; the formula for the resource adaptation model is:

[0022]

[0023] In the formula: f p (x) represents the p-th objective function, and x represents the n-dimensional decision vector, x=[x1,x2,…,x]. n ].

[0024] In a further explanation of the present invention, the SubTask is obtained by data operation mapping between Pods; the Pareto optimal solution is obtained based on the comprehensive calculation and classification results of the subtask size, the existence of alarm data, and network latency.

[0025] As further explained in this invention, the Master manages the port exposure scheduling of Pods through configuration files; the image database is responsible for collecting and storing file information for each version of the Pod.

[0026] The present invention has the following beneficial effects:

[0027] This invention addresses the problem of fault prediction before a Pod exposes its port by using a Markov chain to construct a resource early warning model when a Pod under a Node in the cluster is set to be exposed to the outside, instead of using the Pod with the most idle resources as the exposed port.

[0028] Secondly, in a distributed cluster, data transmission between exposed Pods across multiple clusters is mapped to TaskManagers. Data operations between Pods are mapped to subtasks, and a resource adaptation model combined with Flink's Task Slots (process slots) is used to evenly distribute these subtasks across shared slots. This maximizes the utilization of TaskSlots resources, ensuring that larger subtasks are evenly distributed across TaskManagers. Furthermore, operators with overlapping operations across multiple Pods are chained into the same Subtask, reducing data handover and caching between threads, improving overall throughput, and lowering processing latency. Attached Figure Description

[0029] Figure 1 Flowchart for optimizing Pod scheduling methods.

[0030] Figure 2 A model diagram for optimizing Pod scheduling methods. Detailed Implementation

[0031] The invention will now be further described with reference to the accompanying drawings.

[0032] Example 1:

[0033] A distributed Pod scheduling optimization method based on Kubernetes and Flink includes the following steps:

[0034] Step 1: Create a central Kubernetes-based cluster on the cluster management server;

[0035] Step 2: When preparing to set a Pod under a Node in the cluster to be exposed to the outside world, build a resource early warning model based on Markov chains;

[0036] Step 3: Data transmission between exposed Pods in multiple clusters is mapped to TaskManager. Using the resource adaptation model combined with Flink's Task Slot, SubTasks optimized by the Pareto optimal solution algorithm are sequentially placed into different Task Slots to achieve a uniform distribution of SubTasks on the TaskManager.

[0037] Kubernetes is an open-source Linux container automation platform used to manage multiple hosts forming a cluster. The TaskManager executes data stream tasks or subtasks, caching and interacting with data streams. At least one TaskManager is required. The Task Slot is a process slot, with a default number equal to the number of CPU cores. Subtasks are evenly distributed across TaskManagers with multiple Task Slots, sharing a single process, TCP connections, and heartbeat messages. They can also share datasets and data structures, reducing task overhead. Furthermore, chaining operators with overlapping operations across multiple Pods into the same Subtask reduces data handover and caching between threads, improves overall throughput, and reduces processing latency.

[0038] Example 2:

[0039] A distributed Pod scheduling optimization method based on Kubernetes and Flink includes the following steps:

[0040] Step 1: Create a central Kubernetes-based cluster on the cluster management server;

[0041] The Kubernetes-based cluster mainly includes Master, Node, and Pod, and its analysis methods include:

[0042] Step 1: Deploy the Master on the cluster management server and store the configuration files for the central and local Pods of the distributed cluster.

[0043] Step 2: Obtain historical image data from the image database, including server configuration data for each version of the Pod, and sort the values ​​from largest to smallest to determine the Pod scheduling priority.

[0044] Step 2: When preparing to set a Pod under a Node in the cluster to be exposed to the outside world, build a resource early warning model based on Markov chains;

[0045] The formula for the resource early warning model is:

[0046]

[0047] In the formula: X(k) represents the state vector of the trend analysis and prediction object at time t=k, P represents the one-step transition probability matrix, and X(k+1) represents the state vector of the trend analysis and prediction object at time t=k+1.

[0048] Step 3: Data transmission between exposed Pods in multiple clusters is mapped to TaskManager. Using the resource adaptation model combined with Flink's Task Slot, SubTasks optimized by the Pareto optimal solution algorithm are sequentially placed into different Task Slots to achieve a uniform distribution of SubTasks on the TaskManager.

[0049] The formula for the resource adaptation model is:

[0050]

[0051] In the formula: f p (x) represents the p-th objective function, and x represents the n-dimensional decision vector, x=[x1,x2,…,x]. n ].

[0052] Example 3:

[0053] A distributed Pod scheduling optimization method based on Kubernetes and Flink includes the following steps:

[0054] Step 1: Create a central Kubernetes-based cluster on the cluster management server;

[0055] The Kubernetes-based cluster mainly includes Master, Node, and Pod, and its analysis methods include:

[0056] Step 1: Deploy the Master on the cluster management server and store the configuration files for the central and local Pods of the distributed cluster.

[0057] Step 2: Obtain historical image data from the image database, including server configuration data for each version of the Pod, and sort the values ​​from largest to smallest to determine the Pod scheduling priority.

[0058] Step 2: When preparing to set a Pod under a Node in the cluster to be exposed to the outside world, build a resource early warning model based on Markov chains;

[0059] The formula for the resource early warning model is:

[0060]

[0061] In the formula: X(k) represents the state vector of the trend analysis and prediction object at time t=k, P represents the one-step transition probability matrix, and X(k+1) represents the state vector of the trend analysis and prediction object at time t=k+1.

[0062] By using a two-step transition matrix to generate a dataset and analyzing historical data, it was found that among the current Pod historical image database monitoring metrics, 30% are abnormal and 70% are normal. When preparing to expose the Pod's port, 60% of the abnormal metrics may continue to be abnormal, and 40% of the normal metrics may turn abnormal. At the same time, the 70% of normal monitoring data may still be normal, while 30% may turn abnormal.

[0063] Historical analysis shows anomalies and normality (30%, 70%).

[0064] When the Pod port is about to be exposed, the error turns into a normal operation [0.6, 0.4].

[0065] Normal to abnormal [0.3, 0.7];

[0066] Calculation process: The historical in direction = 0.3 x 0.6 + 0.3 x 0.7 = 0.39;

[0067] Out direction = 0.3x0.4 + 0.7x0.7 = 0.61;

[0068] Ultimately, the probability of failure after the Pod port is exposed is 39%, and the probability of normal operation is 61%.

[0069] Step 3: Data transmission between exposed Pods in multiple clusters is mapped to TaskManager. Using the resource adaptation model combined with Flink's Task Slot, SubTasks optimized by the Pareto optimal solution algorithm are sequentially placed into different Task Slots to achieve a uniform distribution of SubTasks on the TaskManager.

[0070] The formula for the resource adaptation model is:

[0071]

[0072] In the formula: f p (x) represents the p-th objective function, and x represents the n-dimensional decision vector, x=[x1,x2,…,x]. n ].

[0073] For the multi-objective programming optimal solution problem that the resource adaptation model needs to solve, let its variable feasible region be S, and the corresponding objective feasible region be Z=f(S).

[0074] Given a feasible point ,have ,have ,but This is called the absolute optimal solution to a multi-objective programming problem. If it does not exist... , making ,but This is called an efficient solution to the goal programming problem. An efficient solution to a multi-objective programming problem is also called a Pareto optimal solution.

[0075] Example 4:

[0076] The difference between this embodiment and embodiment 3 is that: the SubTask is obtained by data operation mapping between Pods; the Pareto optimal solution is obtained by comprehensive calculation and classification results of the subtask size, the existence of alarm data, and network latency.

[0077] Example 5:

[0078] The difference between this embodiment and embodiment 4 is that the Master manages the port exposure scheduling of the Pod through configuration files; the image database is responsible for collecting and storing file information for each version of the Pod.

[0079] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. The scope of protection of the present invention is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its spirit and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A distributed Pod scheduling optimization method based on Kubernetes and Flink, characterized in that... Includes the following steps: Step 1: Create a central Kubernetes-based cluster on the cluster management server; Step 2: When preparing to set a Pod under a Node in the cluster to be exposed to the outside world, build a resource early warning model based on Markov chains; Step 3: Map the data between exposed Pods in multiple clusters to TaskManager. Using the resource adaptation model combined with Flink's Task Slot, the SubTasks optimized by the Pareto optimal solution algorithm are placed into different Task Slots in sequence, thus achieving a uniform distribution of SubTasks on the TaskManager. The Kubernetes-based cluster mainly includes Master, Node, and Pod, and its analysis methods include: Step 1: Deploy the Master on the cluster management server and store the configuration files for the central and local Pods of the distributed cluster. Step 2: Obtain historical image data from the image database, including server configuration data for each version of the Pod, and sort the values ​​from largest to smallest to determine the Pod scheduling priority; The formula for the resource early warning model is: ； In the formula: X(k) represents the state vector of the trend analysis and prediction object at time t=k, P represents the one-step transition probability matrix, and X(k+1) represents the state vector of the trend analysis and prediction object at time t=k+1; The resource adaptation model is constructed using the Pareto analysis method; the formula for the resource adaptation model is: ； In the formula: f p (x) represents the p-th objective function, and x represents the n-dimensional decision vector, x=[x1,x2,…,x]. n ].

2. The distributed Pod scheduling optimization method based on Kubernetes and Flink according to claim 1, characterized in that: The SubTask is obtained by mapping data between Pods; the Pareto optimal solution is obtained based on the comprehensive calculation and classification results of the subtask size, the existence of alarm data, and network latency.

3. The distributed Pod scheduling optimization method based on Kubernetes and Flink according to claim 1, characterized in that: The Master manages the port exposure scheduling of Pods through configuration files; the image database is responsible for collecting and storing file information for each version of the Pod.

Images