Method, medium and device for pod evictions in k8s clusters

Abstract

The present application relates to the field of pod scheduling, in particular to a pod removal method, medium and equipment in a K8s cluster. It comprises: using a graph neural network, for each target node and each comparison node, respectively extracting features of the corresponding node topology graph to generate a second target node vector and a second comparison node vector; according to the similarity between the second target node vector and the second comparison node vector, and the stability of the CPU occupancy rate in the preset historical period, determining whether the candidate removed pod is retained. In the present application, when constructing the node vector, not only the working mechanism features of the node are extracted according to the communication situation of the node, but also the hardware resource features are extracted according to the hardware resource configuration. Therefore, according to the running stability of the comparison node in the preset historical period, it can be determined whether the candidate removed pod is more likely to affect the stability of the cluster, thereby reducing the situation of mistakenly removing the pod.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of pod scheduling, in particular to a method, medium and device for pod evicting in a K8s cluster. BACKGROUND

[0002] A Kubernetes (often referred to as K8s) cluster is an open-source platform for automating the deployment, scaling, and management of containerized applications. It provides a framework for running distributed systems and ensures high availability, scalability, and efficient use of resources for applications.

[0003] A pod is the smallest deployable unit in K8s, usually containing one or more tightly coupled containers. Applications that implement business functions are typically deployed in the containers corresponding to the pod, so that the pod can implement the corresponding business functions when running on the corresponding node. A node is a worker machine in K8s with certain computing resources (CPU, memory, etc.), which is the part of the cluster that actually performs work and can be a virtual machine or a physical machine. K8s and microservices architecture complement each other, the former provides an ideal running environment and technical support for the latter, enabling enterprises to focus more on the implementation of business logic rather than the management of underlying infrastructure. In existing complex business scenarios, each microservice instance (i.e., a running program copy of a service) of a microservice is usually deployed as one or more pods and runs on different nodes in K8s, which collectively provide the functionality of the microservice to provide higher service stability.

[0004] Generally, in K8s, whether to evict existing pods in a node or create new pods in the node is determined based on whether the current resource occupancy in the node exceeds a pre-set resource occupancy alarm threshold. As business functions develop and upgrade, the resource requirements (especially CPU requirements) of pods corresponding to each business function also change accordingly, causing the resource occupancy of some nodes in the cluster to exceed the threshold and trigger an alarm. In existing technologies, some pods in the node are evicted to release the corresponding resources based on the occupancy of the corresponding resources. This evicting method of pods is rough and prone to mistakenly evicting pods that do not affect the stability of the cluster. SUMMARY

[0005] To solve one of the above technical problems, the technical solution adopted by the present application is as follows:

[0006] According to one aspect of the present application, a method for evicting pods in a K8s cluster is provided, the method comprising the following steps:

[0007] other nodes in the cluster that have deployed the same pod as the candidate evicted pod as comparison nodes of the target node; the target node is a node to which the candidate evicted pod belongs; the candidate evicted pod is any pod in the target node with a CPU occupancy rate ranking in a preset position;

[0008] using a graph neural network, performing feature extraction on a node topology graph corresponding to the target node and each comparison node respectively to generate a first target node vector corresponding to the target node and a first comparison node vector corresponding to each comparison node;

[0009] The nodes in the node topology graph include nodes corresponding to each pod running in the target node and nodes corresponding to pods in other nodes that have communication interaction with each pod in the target node in a preset historical period, and the attribute values of the nodes include a pod ID, a work type number, and a node ID of a node to which the pod belongs. The directed edges in the node topology graph are connected between any two nodes that have communication interaction, the connection direction is the same as the communication direction, and the directed edges are provided with a communication frequency between the two nodes in the corresponding communication direction. The preset historical period is a historical period with a preset length before the alarm of the node to which the evicted pod belongs;

[0010] splicing the first target node vector with a hardware attribute vector of the target node to generate a second target node vector;

[0011] splicing each first comparison node vector with a hardware attribute vector of the corresponding node to generate a corresponding second comparison node vector; the hardware attribute vector of the node includes a CPU attribute, a memory attribute, a storage attribute, a network attribute, a GPU attribute, and a motherboard attribute;

[0012] If the similarity between the second target node vector and any second comparison node vector is greater than a preset similarity threshold, and the node corresponding to the second comparison node vector has a CPU occupancy rate greater than or equal to a first preset occupancy threshold in the preset historical period, the candidate evicted pod is evicted; the first preset occupancy threshold is less than or equal to a CPU usage alarm threshold.

[0013] According to a second aspect of the present application, a non-transitory computer readable storage medium is provided, the non-transitory computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned pod evicting method in a K8s cluster.

[0014] According to a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for removing a pod in a K8s cluster when executing the computer program.

[0015] The present application has at least one of the following beneficial effects:

[0016] In the present application, when a pod is determined as a candidate removal pod, the similarity between the target node where the candidate removal pod is located and the comparison node in the cluster where the same pod as the candidate removal pod has been deployed is calculated to determine whether the current candidate removal pod is more likely to cause instability of the node where it is located, and thus to further determine whether the candidate removal pod should be removed preferentially.

[0017] Meanwhile, when constructing the node vector corresponding to each node (i.e., the second target node vector and the second comparison node vector), not only the working mechanism features of the node are extracted according to the communication situation of each node in the node in a preset historical period before the alarm, but also the hardware resource features of the node are extracted according to the hardware resource configuration information of the node. Therefore, when calculating the similarity, the comparison node with the same running mode as the target node can be found more accurately, and then whether the candidate removal pod is more likely to affect the stability of the cluster can be determined according to the running stability of the comparison node in the preset historical period, thereby reducing the situation of mistakenly removing the pod that does not affect the stability of the cluster. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 A flowchart of a method for removing a pod in a K8s cluster provided by an embodiment of the present application;

[0020] Figure 2 A flowchart of a method for creating a pod in a K8s cluster provided by an embodiment of the present application;

[0021] Figure 3 A flowchart of a compensation scheduling method for a pod in a K8s cluster provided by an embodiment of the present application;

[0022] Figure 4A schematic diagram of a node topology graph of a node provided by an embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative work fall within the protection scope of the present application.

[0024] According to one possible embodiment of the present application, as shown in Figure 1 A method for pod evicting in a K8s cluster is provided, and the method comprises the following steps:

[0025] S100: Other nodes in the cluster, which have deployed the same pod as the candidate evicting pod, are taken as the comparison nodes of the target node. The target node is the node to which the candidate evicting pod belongs. The candidate evicting pod is any pod in the target node with a CPU occupancy rate in the top preset position.

[0026] In a K8s cluster, multiple nodes are set to deploy and run different pods to realize various business functions (i.e., multiple microservices). For example, in a flight ticket purchase scenario, ticket information query, seat selection, ticket purchase payment, change and refund, etc. are all different microservices. Taking the microservice corresponding to the ticket information query as an example, multiple identical pods are usually set to jointly implement this function in order to cope with the access volume during the ticket query peak time, and different pods are deployed on different nodes in the cluster for running. According to a preset load balancing strategy, different pods can share a corresponding degree of access volume, and at the same time, the pods are deployed on different nodes. If a node (i.e., node) fails, the replicas on other nodes can still continue to provide services, thereby improving the reliability of the service.

[0027] In actual operation process, with the development and upgrading of business functions, not only more new pods will be established in the node, but also the original pods will be upgraded, thereby causing the corresponding changes in resource requirements (especially the demand for CPU). In the prior art, a CPU usage alarm threshold (such as 80%) is usually set. Once the actual CPU usage of the node is greater than the CPU usage alarm threshold, the node will be in an alarm state, and at this time, part of the pods in the node need to be removed (that is, evicted). Usually, a pre-set eviction strategy (such as removing the pods ranked in the top pre-set positions in terms of resource occupation or removing the pods with a resource occupation rate exceeding a pre-set threshold) is used to remove part of the pods in the node as candidate removal pods, and then the pods are evicted.

[0028] Since in the same cluster, usually one microservice will set multiple identical pods, that is, there will be replica pods identical to the current candidate removal pod. And when each pod is deployed and scheduled, since its screening strategy is roughly the same, the pods will be mounted on different nodes with high similarity. Therefore, in this step, after the candidate removal pod is determined, the node to which the other replica pods belong is used as the comparison node.

[0029] Specifically, before S100, the method further includes the following method for determining the candidate removal pod:

[0030] S101: When the CPU resource occupation rate of any node in the cluster is greater than or equal to the CPU usage alarm threshold, the pod with a CPU resource occupation rate greater than a pre-set eviction threshold in the node is used as the candidate removal pod. The pre-set eviction threshold is less than the CPU usage alarm threshold. Specifically, the pre-set eviction threshold can be 15%, and the CPU usage alarm threshold can be 80%.

[0031] S200: Using a graph neural network, the node topology graph corresponding to each of the target node and the comparison node is respectively subjected to feature extraction, thereby generating a first target node vector corresponding to the target node and a first comparison node vector corresponding to each of the comparison nodes.

[0032] For example, Figure 4As shown, the nodes in the node topology graph include nodes corresponding to each pod running in the target node in a preset historical period, such as d1, d2, and d3 in the figure, and nodes corresponding to pods in other nodes that have communication interaction with each pod in the target node, such as a1, a2, b1, c1, and c2 in the figure. The attribute values of the nodes include pod ID, work type number, and node ID of the node to which the pod belongs. Specifically, the node attribute value can be in the form of (d1, 05, D), where d1 is the pod ID, 05 is the work type number, and D is the node ID of the node to which the pod belongs. The directed edges in the node topology graph are connected between any two nodes that have communication interaction, the connection direction is the same as the communication direction, and the communication frequency between the two nodes in the corresponding communication direction is set on the directed edge. The preset historical period is a period from the time when the alarm of the node to which the pod belongs is removed to an earlier historical time. For example, it can be a historical period corresponding to the previous 10 minutes from the time when the alarm of the node to which the pod belongs is removed. For example, the time corresponding to the node alarm is 10:00:00, and the corresponding preset historical period is 9:50:00 to 10:00:00.

[0033] Generally, a rapid increase in CPU occupancy in a node is probably caused by an increase in access traffic of multiple pod corresponding services, which leads to an increase in CPU usage of the pod. Correspondingly, the increase in access traffic of multiple services is usually not sudden, but gradually increases in a continuous period, which leads to an increase in CPU occupancy of the pod corresponding to the service until the CPU resource occupancy rate of the node is greater than or equal to the CPU usage alarm threshold and an alarm is issued. Therefore, in this embodiment, when the node topology graph is constructed, the communication topology of each pod in the node in the preset historical period is used. Specifically, the node topology graph can reflect the communication topology structure and the related characteristics of the communication volume of each pod in the node, which indicate the working condition of the node in the period of increasing access traffic.

[0034] Graph Neural Networks (GNN) is a deep learning model specially used for processing graph structure data. In the topology graph, nodes can represent entities (in this embodiment, representing pod ID, work type number, node ID of the node to which the pod belongs, etc.), and edges represent the relationship between these entities (in this embodiment, representing the communication direction and the communication frequency between two nodes). GNN extracts features by directly operating on the graph structure. Therefore, after the directed node topology graph corresponding to the node is constructed in this embodiment, GNN can be used to extract the feature vector of the graph, that is, the first target node vector and the first comparison node vector in this embodiment.

[0035] S300: Concatenate the first target node vector with the hardware attribute vector of the target node to generate a second target node vector.

[0036] S400: Concatenate each first comparison node vector with the hardware attribute vector of the corresponding node to generate a corresponding second comparison node vector. The hardware attribute vector of the node includes CPU attributes, memory attributes, storage attributes, network attributes, GPU attributes, and motherboard attributes. The above specific attribute values can form specific element values in the hardware attribute vector through existing encoding methods. For example, if the number of CPU cores in the CPU attribute is 6, the element value in the corresponding dimension of the hardware attribute vector is 6.

[0037] Specifically, the CPU attributes can include the number of cores, architecture type, and working frequency of each CPU core; the memory attributes can include the total amount of physical memory; the storage attributes can include the total amount of hard disk storage and the type of hard disk such as HDD (mechanical hard disk), SSD (solid state disk), NVMe, and different types; the network attributes can include the interface type, maximum transmission rate (i.e. bandwidth) of the network interface, and average communication delay; the GPU attributes can include the number of GPUs, the model of each GPU, the video memory capacity of the GPU, and the working frequency of the GPU; and the motherboard attributes can include the motherboard model and manufacturer. Of course, other hardware attributes such as cooling system attributes and power supply attributes can also be added.

[0038] By concatenating the above vectors reflecting the working conditions of the node and the hardware attribute vectors, the characteristics of a node can be more comprehensively and accurately reflected. Therefore, in this embodiment, the similarity (such as cosine similarity) between the node vectors corresponding to two nodes can be calculated to reflect the similarity between the two nodes.

[0039] S500: If the similarity between the second target node vector and any second comparison node vector is greater than a preset similarity threshold, and the CPU occupancy rate of the node corresponding to the second comparison node vector in a preset historical period is greater than or equal to a first preset occupancy threshold, the candidate removal pod is removed; the first preset occupancy threshold is less than or equal to the CPU usage alarm threshold. For example, the first preset occupancy threshold can be 70%.

[0040] S600: If the similarity between the second target node vector and any second comparison node vector is greater than a preset similarity threshold, and the CPU occupancy rate of the node corresponding to the second comparison node vector in a preset historical period is always less than the first preset occupancy threshold, the candidate removal pod is retained.

[0041] In the embodiment, S500 and S600 can select the comparison nodes with high similarity to the target node in the K8s cluster by performing the similarity comparison between the target node and each comparison node. Then, whether the comparison nodes are stable in the preset historical period is determined according to whether the CPU occupancy rate of the comparison nodes is less than the first preset occupancy threshold in the preset historical period. Then, whether the candidate removal pod can be stably run in the target node is further determined according to the running stability of the comparison nodes, so as to determine whether the candidate removal pod is expelled, thereby reducing the situation of mistakenly removing the pod that does not affect the stability of the cluster.

[0042] In addition, in order to further improve the accuracy of the determination, S500 can be replaced by: if the stability determination value P of the candidate removal pod is greater than Y1, the candidate removal pod is retained. Y1 is a determination threshold, P=P1 / P2, P1 is the number of second comparison node vectors with similarity greater than a preset similarity threshold between the second comparison node vector and the candidate removal pod, and the CPU occupancy rate of the corresponding node is less than the first preset occupancy threshold in the preset historical period, and P2 is the number of second comparison node vectors with similarity greater than the preset similarity threshold between the second comparison node vector and the candidate removal pod.

[0043] As another possible embodiment of the present application, the pod removal method in the K8s cluster further comprises:

[0044] S710: if all the candidate removal pods in the target node are retained, the comparison node with the maximum similarity corresponding to each candidate removal pod and the target node are sequentially compared in the order of the CPU occupancy rate of each candidate removal pod from large to small, to determine the pod to be removed in the target node.

[0045] The comparison processing comprises:

[0046] S711: obtaining the pod set A of the comparison node corresponding to the maximum similarity corresponding to the candidate removal pod.

[0047] S712: obtaining the pod set B of the target node.

[0048] S713: the pod existing in B and not existing in A is taken as a pre-removal pod.

[0049] If all the candidate removal pods are retained after the determination of S500 and S600, it indicates that the current determined candidate removal pods can be stably run in the corresponding other replica pods in the corresponding node, and thus it can be indicated that the current candidate removal pods are not the main reason for the rapid over-standard of the CPU occupancy rate of the node.

[0050] Therefore, in the embodiment, the pod existing in B and not existing in A is taken as the pre-removal pod, and the pod existing in the target node and not existing in the maximum similar corresponding degree corresponding to the candidate removal pod is taken as the main reason for the rapid over-standard of the CPU occupancy rate of the node.

[0051] After the pre-removal pods are determined, the determined pre-removal pods are removed, and the present application provides two parallel pod removal embodiments.

[0052] Pod removal embodiment one

[0053] S714: The pre-removal pods are removed in turn according to the order from small to large of the CPU occupancy rate.

[0054] S715: After each pre-removal pod is removed, if the CPU usage rate of the target node is still in the alarm state, the next pre-removal pod is continuously removed.

[0055] S716: After each pre-removal pod is removed, if the CPU usage rate of the target node is in the non-alarm state, the comparison processing is stopped.

[0056] The present embodiment is suitable for the case that the total number of candidate removal pods is small, for example, there are only two candidate removal pods in total, at this time, all the candidate removal pods can be compared and processed, and then all the pre-removal pods are determined and removed in turn. Since the total number of candidate removal pods is small, even if the full comparison processing is performed, it will not take too much time.

[0057] Pod removal embodiment two

[0058] S724: If the sum of the CPU occupancy rates of all the currently determined pre-removal pods is greater than the total excess amount of the current CPU usage rate of the target node, the comparison processing of the next candidate removal pod is stopped. The total excess amount of the current CPU usage rate of the target node is the difference between the current CPU usage rate of the target node and the second preset occupancy threshold, and the second preset occupancy threshold is less than the first preset occupancy threshold. The second preset occupancy threshold can be 65%.

[0059] S725: If the sum of the CPU occupation rates of all the pre-removal pods currently determined is less than the total amount of the excess of the current CPU usage of the target node, continue to perform the comparison process on the next candidate removal pod.

[0060] In the embodiment, the comparison process is performed on each candidate removal pod. After the comparison process, the size between the sum of the CPU occupation rates of all the pre-removal pods currently determined and the total amount of the excess of the current CPU usage of the target node is compared, so that the target node can be released from the alarm state as soon as possible, and the comparison process can be performed on all the candidate removal pods to reduce the time consumption of the comparison process.

[0061] As another possible embodiment of the present application, as shown in Figure 2 A method for creating a pod in a K8s cluster is also provided, and the method comprises the following steps:

[0062] A100: Match the node vector of the node to be mounted corresponding to the newly created pod with each abnormal node vector of the microservice corresponding to the newly created pod. The abnormal node vector is the node vector of the node after the pod belonging to the same microservice as the newly created pod is removed. The node vector comprises a feature vector of communication interaction of each pod included in the node in a preset historical period and a hardware attribute vector of the node. The preset historical period is a period from when the node of the removed pod alarms to an earlier historical time.

[0063] Specifically, the abnormal node vector of the microservice is obtained according to the following steps:

[0064] A101: When the pod is removed from the node, the microservice label corresponding to the removed pod is obtained.

[0065] A102: Obtain the node vector corresponding to the node after the pod is removed.

[0066] A103: According to the microservice label, the node vector is taken as the abnormal node vector of the corresponding microservice.

[0067] Since there are multiple pods under the same microservice, which are respectively deployed on different nodes in the cluster. Therefore, once the CPU demand of the pod does not match the CPU supply of the current node, the pod will not be able to continue to run stably in the node, that is, the node will issue an alarm, and the pod will be removed from the current node.

[0068] In this embodiment, the node vector of the node to which the pod belongs before the pod is removed is obtained as the abnormal node vector corresponding to the removed pod. Each abnormal node vector in this embodiment is a node vector that does not contain the same pod as the removed pod. Specifically, when constructing the node topology graph corresponding to the node, the topology information related to the same pod as the removed pod in the topology graph can be deleted before feature extraction is performed. For example, if d3 in FIG. 8 is the same pod as the removed pod, d3, a2, and c2 and the edges therebetween can be deleted. Figure 4

[0069] Thus, with the accumulation of pod removal work in daily life, a large number of abnormal node vectors corresponding to each microservice can be obtained, and these vectors indicate that the microservice cannot be stably run on the nodes corresponding to the abnormal node vectors. Therefore, in A100, whether the node to be mounted is a node more suitable for deploying a new pod can be determined by matching the node vector of the current node to be mounted with each abnormal node vector corresponding to the microservice.

[0070] Further, the abnormal node vector in this embodiment can also be obtained after the pod to be removed is determined by the above-mentioned pod removal method in the K8s cluster.

[0071] The node vector is obtained according to the following steps:

[0072] A112: Obtain the node topology graph of the node to be processed. The nodes in the node topology graph include the nodes corresponding to each pod running in the node to be processed and the nodes corresponding to the pods in other nodes that have communication interaction with each pod in the node to be processed in a preset historical period. The attribute values of the nodes include pod ID, work type number, and node ID of the node to which the pod belongs. The directed edges in the node topology graph are connected between any two nodes that have communication interaction, the connection direction is the same as the communication direction, and the directed edges are provided with the number of communications between the two nodes in the corresponding communication direction.

[0073] A122: Use a graph neural network to perform feature extraction on the node topology graph to generate a first target node vector corresponding to the node to be processed.

[0074] A132: Concatenate the first target node vector with the hardware attribute vector of the node to be processed to generate a node vector corresponding to the node to be processed.

[0075] The steps of obtaining the node vector in this embodiment are the same as the ways of obtaining the node vector in S200 and S300 described above, and the only difference is that the pods included in the node are different.​

[0076] A200: If the matching fails, the newly created pod is deployed on the node to be mounted for running.

[0077] A300: If the matching succeeds, the node to be mounted is determined as a node unsuitable for running the newly created pod.

[0078] In the vector matching in the above steps, the cosine similarity of the calculated vectors can be used, and then compared with the set threshold to determine whether the matching succeeds.

[0079] In the embodiment, after the node to be mounted corresponding to the newly created pod is determined, the node to be mounted is further judged. Specifically, the similarity between the node vector of the node to be mounted and the node vector of the node where the same pod is removed when the same pod is removed from the newly created pod is used to determine the similarity between the node to be mounted and the node where the same pod is removed in the historical rescheduling. Since the node vector includes the feature vector of the communication interaction of each pod in the node in the preset historical period when the pod is removed before, and the hardware attribute vector of the node, the node vector can reflect the running mode corresponding to the node. In the present application, the similarity matching between the node vectors can avoid the newly created pod being deployed again in the node having the same running mode as before in the rescheduling, and further avoid the unstable running condition of the resource occupation alarm of the node after the pod is newly created.

[0080] A400: If the number of consecutive times of matching failure of any abnormal node vector corresponding to the microservice is greater than the preset matching number threshold, the abnormal node vector is deleted.

[0081] In this step, the historical abnormal node vector corresponding to the microservice can be cleaned in time. Since the responding pod in the node running unstably in the history is removed and a new more suitable node is mounted for running as the cluster is continuously scheduled, the node vector corresponding to the node also changes as the scheduling is performed, and thus the difference between the topology structure of the node in the current cluster and the topology structure of the node corresponding to the abnormal node vector too early before increases. Therefore, the abnormal node vector stored too early before has no use value and should be deleted. In this step, whether the abnormal node vector has use value is determined by the number of consecutive times of matching failure of the abnormal node vector, and whether the abnormal node vector is cleaned is determined. Thus, by deleting the useless abnormal node in time, the remaining abnormal node vectors can be more useful vectors, and thus the effectiveness of the matching operation is improved.

[0082] As another possible embodiment of the present application, as Figure 3As shown, a method for compensatory scheduling of a pod in a K8s cluster is also provided, and the method comprises the following steps:

[0083] B100: According to the number of pre-warning nodes corresponding to the node to which the removed pod belongs in the cluster, generate the same number of compensation pods under the microservice corresponding to the removed pod. The compensation pods are the same as the removed pods.

[0084] The pre-warning node is a node that has a similarity greater than a first similarity threshold to the node to which the removed pod belongs, has a CPU occupancy rate greater than a first preset occupancy threshold in a preset historical period, and runs a pod identical to the removed pod. Specifically, the preset historical period is a period from the time when the node to which the removed pod belongs is warned to an earlier historical time.

[0085] Since, in consideration of load balancing and ensuring service high availability and fault tolerance, multiple identical pod instances of the same microservice are usually run on different node nodes. Moreover, since multiple pods select nodes according to substantially the same screening condition, the nodes corresponding to the multiple pods have a high similarity. Thus, in the present embodiment, when a pod is removed, nodes with a high similarity to the node to which the removed pod belongs are found, and the number of pre-warning nodes is determined according to the stability performance of these high-similarity nodes in the preset historical period, i.e., whether there is a situation where the CPU occupancy rate is greater than the first preset occupancy threshold in the preset historical period. Thus, the number of determined pre-warning nodes is the number of other pods running unstably under the microservice corresponding to the removed pod, i.e., the matching degree of these pods to the current node is not high, and these pods are more likely to be removed due to CPU occupancy warning in subsequent operation.

[0086] Further, in the present embodiment, before B100, the method further comprises:

[0087] B110: Obtain the node vector corresponding to the node to which the removed pod belongs.

[0088] B120: If any other node running a pod identical to the removed pod in the cluster has a CPU occupancy rate greater than a first preset occupancy threshold in a preset historical period, the other node is determined as an initial node. The first preset occupancy threshold is less than or equal to the CPU usage warning threshold.

[0089] Generally, the first preset occupation threshold is less than the CPU usage alarm threshold, and is very close to the CPU usage alarm threshold, for example, the CPU usage alarm threshold is 80%, and the first preset occupation threshold is 75%. If, in the preset period, other nodes running the same pod as the removed pod exist, and the CPU occupation rate is greater than the first preset occupation threshold, it indicates that the running of the same pod as the removed pod on these nodes is unstable, and these nodes also have a certain alarm risk, which may soon alarm in the subsequent use process.

[0090] B130: If the similarity between the node vector corresponding to the node to which the removed pod belongs and the node vector corresponding to each initial node is greater than the first similarity threshold, the initial node is determined as the pre-alarm node corresponding to the node to which the removed pod belongs.

[0091] In this step, the similarity between the working conditions of two nodes is calculated by the node vector which can reflect the communication topology characteristics and hardware attribute characteristics of the node, and then the pre-alarm node can be more accurately selected from the initial node.

[0092] In this embodiment, when determining the number of pre-alarm nodes corresponding to the node to which the removed pod belongs, the running stability of the node in the preset historical period is first used for rapid preliminary screening, and the corresponding step is B120. Since this method only needs to be completed by threshold comparison, the screening speed can be improved. Then, on the basis of preliminary screening, the node vector of the related node is obtained for vector comparison for fine screening, and the corresponding step is B130. Therefore, by rapid preliminary screening first, the number of nodes in subsequent fine screening is reduced, and the workload of vector matching is reduced.

[0093] Specifically, the node vector in B130 of the embodiment is obtained according to the performance of each node in the preset historical period, and then the similarity between the node vectors is calculated as the similarity between any two nodes. The node vector corresponding to each node in the preset period can be obtained according to the steps of S200, S300 and S400 described above, and will not be repeated here.

[0094] B200: From the plurality of candidate nodes corresponding to the compensation pod, determine the target mounting node corresponding to the compensation pod.

[0095] The target mounting node is a candidate node with a similarity to the node to which the removed pod belongs less than a second similarity threshold and a CPU occupancy less than a second preset occupancy threshold in a preset historical period. The second similarity threshold is less than the first similarity threshold. The second preset occupancy threshold is less than the first preset occupancy threshold. The second preset occupancy threshold can be 65%.

[0096] Before B200, the method further includes:

[0097] B210: selecting a plurality of initial nodes from the cluster according to the resource configuration requirements of the compensation pods and the preset constraint conditions.

[0098] B220: prioritizing the initial nodes according to resource utilization, load balancing, and node affinity of the initial nodes.

[0099] B230: determining the initial nodes in a preset interval as candidate nodes corresponding to the compensation pods.

[0100] B300: deploying the compensation pods to the corresponding target mounting nodes to run, and removing the pods in the pre-warning nodes that are the same as the removed pods.

[0101] In this embodiment, the same number of compensation pods are generated according to the number of pre-warning nodes, and a new node is selected for each compensation pod as a target mounting node. Since the similarity between the target mounting node and the node to which the removed pod belongs is small, and the running in the preset historical period is more stable, the running of the compensation pods on the target mounting node can also be more stable and reliable. After the compensation pods are deployed to the corresponding target mounting nodes to run, the pods in the pre-warning nodes that are the same as the removed pods are removed.

[0102] Therefore, when the pod is removed, the number of other unstable pods under the corresponding microservice is determined according to the similarity between the removed pod and the node to which the removed pod belongs and the stability performance in the preset historical period. Then the same number of compensation pods are established in advance, and after the compensation pods are deployed to other more matched nodes, the pods in the pre-warning nodes that are the same as the removed pods are removed. Thus, the problem that the business function corresponding to the pod cannot be used during the process from removal to redeployment is avoided, the high availability of the corresponding microservice is ensured, and the user experience of using the business function is improved.

[0103] Moreover, although individual steps of the methods in the present disclosure are described in a particular order in the figures, this is not required or implied as to the order of execution of the steps, nor is it required that all of the steps be executed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into a single step, a single step can be broken into multiple steps, etc.

[0104] From the above description of the embodiments, those skilled in the art will easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to make a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) execute the method according to the embodiments of the present disclosure.

[0105] In the example embodiments of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0106] Those skilled in the art can understand that each aspect of the present disclosure can be implemented as a system, a method or a program product. Therefore, each aspect of the present disclosure can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be generally referred to as a "circuit", a "module" or a "system".

[0107] The electronic device according to this embodiment of the present disclosure. The electronic device is merely an example, and should not bring any limitation to the function and use range of the embodiments of the present disclosure.

[0108] The electronic device is in the form of a general computing device. The components of the electronic device can include, but are not limited to, the at least one processor described above, the at least one memory described above, and a bus connecting different system components (including the memory and the processor).

[0109] The memory stores program codes which can be executed by the processor, so that the processor executes the steps according to various example embodiments of the present disclosure described in the "example method" section of the present specification.

[0110] The memory can include a readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory, and can further include a read-only memory (ROM).

[0111] The storage may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0112] The bus may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0113] The electronic device may also communicate with one or more external devices (e.g., a keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., a router, modem, etc.). This communication may occur via an input / output (I / O) interface. Furthermore, the electronic device may communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0114] Through the description of the above embodiments, it will be readily understood by those skilled in the art that the example embodiments described herein can be implemented via software or via a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or mobile hard drive) or on a network and includes several instructions for enabling a computing device (such as a personal computer, server, terminal device, or network device) to execute the methods according to the embodiments of the present disclosure.

[0115] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which is stored a program product capable of implementing the methods described above. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product comprising program code that, when executed on a terminal device, causes the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0116] A program product can take any combination of one or more computer-readable media. The computer-readable media can be a computer-readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0117] The computer-readable signal medium can include a computer-readable storage medium that is propagated as a carrier wave in a baseband or propagated as part of a carrier wave of a propagated signal in a baseband. The propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport programming for use by or in connection with an instruction execution system, apparatus, or device.

[0118] The program code embodied on the computer-readable media can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the above.

[0119] Program code embodied on the computer-readable media can be in any combination of one or more programming languages. Programming languages include, but are not limited to, object-oriented programming languages (e.g., Java, C++, etc.), and conventional procedural programming languages (e.g., the "C" programming language or similar programming languages). The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device (e.g., through the Internet using an Internet Service Provider). The application program interface (API) can be used to communicate between the user's computing device and the remote computing device.

[0120] In addition, the above-described flowcharts are merely illustrative of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to limit the present application. It is readily understood that the processes shown in the above-described flowcharts do not indicate or limit the time sequence of the processes. In addition, it is readily understood that the processes can be executed synchronously or asynchronously, for example, in a plurality of modules.

[0121] It should be noted that, although several modules or units of the devices for action execution are mentioned in the foregoing detailed description, such division into modules or units is not mandatory. Indeed, according to an embodiment of the present disclosure, features and functions of two or more of the above-described modules or units can be embodied in one module or unit. Conversely, features and functions of one module or unit described above can be further divided into plural modules or units.

[0122] The above merely shows the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed by the present application can be easily conceived by those skilled in the art, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for removing a pod in a K8s cluster, characterized in that: The method comprises the following steps: Use other nodes in the cluster that have deployed the same pod as the candidate pod as the comparison node of the target node; the target node is the node to which the candidate pod belongs; The candidate pods to be removed are any pods with the highest CPU usage ranking in the target node. Using a graph neural network, feature extraction is performed on the node topology graphs corresponding to the target node and each comparison node, generating a first target node vector corresponding to the target node and a first comparison node vector corresponding to each comparison node; The nodes in the node topology graph include nodes corresponding to each pod running on the target node during a preset historical period, and nodes corresponding to pods on other nodes that have communication interactions with each pod on the target node. The attribute values ​​of the nodes include pod ID, work type number, and node ID of the node to which the pod belongs. A directed edge in the node topology graph connects any two nodes that have communication interactions, with the connection direction being the same as the communication direction. The directed edge is set with the number of communications between the two nodes in the corresponding communication direction. The preset historical period is the historical period of the preset length before the alarm of the node to which the pod belongs is removed; Splicing the first target node vector with the hardware attribute vector of the target node to generate a second target node vector; Each first comparison node vector is concatenated with the hardware attribute vector of the corresponding node to generate a corresponding second comparison node vector; the hardware attribute vector of the node includes the node's CPU attributes, memory attributes, storage attributes, network attributes, GPU attributes, and motherboard attributes; If the similarity between the second target node vector and any second comparison node vector is greater than a preset similarity threshold, and the CPU occupancy rate of the node corresponding to the second comparison node vector is greater than or equal to a first preset occupancy threshold in a preset historical period, the candidate pod is removed; The first preset occupancy threshold is less than or equal to the CPU usage alarm threshold; The method further comprises: If all candidate pods are retained in the target node, the comparison node with the maximum similarity corresponding to each candidate pod is compared with the target node in descending order of CPU usage to determine the pod to be removed from the target node. The comparison process includes: Get the pod set A of the comparison nodes corresponding to the maximum similarity of the candidate pods to be removed; Get the pod set B of the target node; Pods that exist in B but not in A are treated as pre-removed pods.

2. The method according to claim 1, characterized in that The method further comprises: If the similarity between the second target node vector and any second comparison node vector is greater than a preset similarity threshold, and the CPU usage of the node corresponding to the second comparison node vector is always less than a first preset usage threshold in a preset historical period, the candidate pod is retained.

3. The method according to claim 1, characterized in that After determining the pod to be removed, the method further includes: Remove the pre-removed pods in ascending order of CPU usage. After removing a pre-removed pod, if the CPU usage of the target node is still in the alarm state, the next pre-removed pod will be removed.

4. The method according to claim 1, wherein After determining the pod to be removed, the method further includes: Remove the pre-removed pods in ascending order of CPU usage. After each pre-removal pod is removed, if the CPU usage of the target node is in a non-alarm state, the comparison process is stopped.

5. The method according to claim 1, wherein After comparing the comparison node with the maximum similarity corresponding to the current candidate pod to be removed with the target node, the method further includes: If the sum of the CPU usage of all currently determined pods to be removed is greater than the total excess CPU usage of the target node, the comparison process for the next candidate pod to be removed is stopped; the total excess CPU usage of the target node is the difference between the current CPU usage of the target node and a second preset usage threshold, where the second preset usage threshold is less than the first preset usage threshold.

6. The method according to claim 5, characterized in that After comparing the comparison node with the maximum similarity corresponding to the current candidate pod to be removed with the target node, the method further includes: If the total CPU usage of all currently determined pre-removal pods is less than the total excess CPU usage of the target node, the comparison process continues with the next candidate pod to be removed.

7. The method according to claim 1, characterized in that Before using other nodes in the cluster that have deployed the same pod as the candidate pod as a comparison node for the target node, the method further includes: When the CPU resource usage of any node in the cluster is greater than or equal to the CPU usage alarm threshold, pods whose CPU resource usage in the node is greater than the preset eviction threshold are selected as candidates for pod removal; the preset eviction threshold is less than the CPU usage alarm threshold.

8. A non-transitory computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements a method for removing a pod in a K8s cluster as described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements a method for removing a pod in a K8s cluster as described in any one of claims 1 to 7.

Images