Method and device for processing Pod service abnormal endpoint after node failure

Abstract

The invention discloses a method and device for processing a Pod service abnormal endpoint after a node fault, and belongs to the technical field of cloud computing container arrangement, and the method comprises the steps: collecting the state data of a node in a Kubernetes cluster after the fault, and storing the state data in an ETCD database; processing the state data of the nodes in the Kubernetes cluster, screening out the abnormal end points of the Kubernetes service, and storing the data of the abnormal end points of the Kubernetes service into an ETCD database; and access control for preventing abnormal updating and abnormal recovery of the Kubernetes service abnormal endpoint is carried out on the Kubernetes service abnormal endpoint. According to the method, the abnormal state of the node in the Kubernetes cluster can be found, the endpoint information associated with the abnormal Pod on the abnormal node is updated in time, the failure recovery efficiency of the Kubernetes cluster is improved, and the user experience is further improved.

Description

Technical Field

[0001] The present invention relates to a method and a device for processing abnormal endpoints of a Pod service after a node failure, and belongs to the technical field of cloud computing container orchestration. Background Art

[0002] In a container cloud environment, when a node fails, Kubernetes uses the node-monitor-grace-period parameter by default to control the processing time after the node fails. By default, the time is 40 seconds, which means that Kubernetes will mark the node as NotReady only after the node has not reported its health status to the API Server for more than 40 seconds. Before that, the node is still considered normal and all Pods associated with it are still considered available.

[0003] Since the load balancing mechanism of Kubernetes services depends on the health status of nodes and Pods, the endpoint list of the service will not be updated immediately before the node is marked as NotReady, and the health check service of the application cannot quickly handle the failure in the case of node abnormality. Therefore, before the failed node is determined to be unavailable, the traffic may continue to be distributed to the Pod on the node, resulting in unresponsive requests or access failures. This situation is particularly serious if the application has high requirements for high availability or is load sensitive, and short-term service interruptions will directly affect the user experience. During this waiting time (usually 40 seconds), since the service cannot promptly identify the failed node and remove the corresponding Pod from the load balancing pool, part of the traffic will be distributed to the failed node, increasing the probability of access failure, resulting in a decrease in the overall availability of the application and an increase in the failure rate of user requests.

[0004] The node-monitor-grace-period parameter controls how long a node will be marked as unavailable after it fails to send a heartbeat signal to the API Server. Appropriately shortening the node-monitor-grace-period parameter can solve this problem, but shortening the duration of this parameter will have some potential negative effects on the Kubernetes cluster. Too short a time can easily lead to misjudgment due to temporary network problems or short-term high load, causing the node to be marked as unavailable prematurely. If the network environment in the cluster is not stable, misjudgment problems may occur frequently. When nodes are frequently marked as unavailable, the Kubernetes controller will start scheduling the affected Pods on other nodes, which will increase the burden on scheduler and controller management. Frequent scheduling will also affect the overall stability and performance of the cluster, especially in large clusters, which may lead to waste of resources. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a method and device for processing abnormal endpoints of Pod services after node failure, which can quickly detect and respond to node abnormalities.

[0006] The technical solution adopted by the present invention to solve the technical problem is: In a first aspect, an embodiment of the present invention provides a method for processing abnormal endpoints of a Pod service after a node failure, comprising the following steps: Step S1, collect the status data of the nodes in the Kubernetes cluster after the failure and store it in the ETCD database; Step S2, processing the status data of the nodes in the Kubernetes cluster, filtering out abnormal Kubernetes service endpoints, and storing the abnormal Kubernetes service endpoint data in the ETCD database; Step S3, performing admission control on abnormal endpoints of Kubernetes services to prevent abnormal updates and abnormal recovery.

[0007] As a possible implementation of this embodiment, step S1, collecting the status data of the nodes in the Kubernetes cluster after the failure and storing it in the ETCD database, includes: Step S101, obtaining a list of node leases in the Kubernetes cluster through Kubernetes authentication; Step S102, calculating the renewal time of the node lease, and calculating the node status through the renewalTime (lease renewal time) field; Step S103, calculating the optimal abnormal waiting time of each node according to the renewal time of the node lease; Step S104, record the node status and store it in the ETCD database.

[0008] As a possible implementation of this embodiment, the step S102, calculating the renewal time of the node lease and calculating the node status through the renewalTime field, includes: Traverse all node lease lists and obtain node lease renewalTime; Calculate the difference between the node lease renewal time and the current time and convert it into milliseconds. The difference is the node lease renewal time. If the node lease renewal time is less than or equal to the optimal abnormal waiting time value of the node in memory, the node is judged to be in a normal state. If the node lease renewal time is greater than the optimal abnormal waiting time value of the node in memory, the node is judged to be in an abnormal state.

[0009] As a possible implementation of this embodiment, the step S103, calculating the optimal abnormal waiting time of each node according to the renewal time of the node lease, includes: At 00:00:00 every day in the memory, initialize an Int (integer) type one-dimensional array for each node, and store the calculated node lease renewal time in the one-dimensional array; At 00:00:00 every day, the values ​​in the Int one-dimensional array of nodes of the previous day are sorted from small to large, and the 98% number of the sorted array is taken using the upward modulus method. This number is used as the optimal exception waiting time of the current node and recorded in the memory Map. The Int one-dimensional array of nodes of the previous day is released.

[0010] As a possible implementation of this embodiment, step S2, processing the status data of the nodes in the Kubernetes cluster, screening out abnormal endpoints of the Kubernetes service, and storing the abnormal endpoint data of the Kubernetes service in the ETCD database, includes: Step S201, traverse the node status list, and if there is an abnormal node, obtain the Pod list with the specified annotation on the abnormal node; Step S202, traverse the Pod list to detect whether the Pod IP is reachable. If the Pod IP is not reachable, record the Pod in the abnormal Pod list; Step S203, traverse the IP unreachable Pod list generated in the previous step, and obtain the corresponding Service list according to the label; Step S204, traverse the generated Service list and delete abnormal Pod IP information in the endpoint; Step S205, record abnormal node information, abnormal endpoint information and abnormal Pod IP information, and store them in the ETCD database.

[0011] As a possible implementation of this embodiment, step S3, performing admission control on abnormal endpoints of Kubernetes services to prevent abnormal updates and abnormal recovery thereof, includes: Step S301, using the Kubernetes admission control ValidatingAdmissionWebhook application to monitor the update of the endpoint; Step S302: judging whether the endpoint of the arrival event has an associated fault node according to the abnormal endpoint information recorded in the ETCD database; Step S303, obtaining the current state of the associated faulty node according to the node state information recorded in the ETCD database, if the current state of the node is normal, deleting the abnormal endpoint information in the ETCD database; if the current state of the node is abnormal, executing the next step; Step S304: According to the faulty Pod IP list recorded in the ETCD database, determine whether the IP updated by the current endpoint update method is a faulty IP. If it is a faulty Pod IP, reject the request.

[0012] As a possible implementation of this embodiment, step S302 includes: Use the Namespace+Name of the endpoint that triggers the update event as the key value to query the endpoint list / registry / node-detect / endpoint in the ETCD database. If the key value does not exist, the process ends. If the key value exists, get the corresponding value and split the value according to the character '-' to get a values ​​array. The first element of the values ​​array is the node name, and the second element is the abnormal Pod IP list; Use the node name as the key value to query the node status in the node status list / registry / node-detect / node-status of the etcd database.

[0013] As a possible implementation of this embodiment, step S303 includes: If the obtained associated node status is normal, delete the record corresponding to the endpoint Namespace+Name as the key value in the endpoint list / registry / node-detect / endpoint, and the process ends; If the status of the associated node obtained in the step is abnormal, get the second element of the values ​​array, which is the abnormal PodIP list string.

[0014] As a possible implementation of this embodiment, step S304 includes: Use the character ';' to cut the abnormal Pod IP list string to get the abnormal IP list; Get the Pod IP value corresponding to the current endpoint resource update operation. If the Pod IP in the updated value is not in the abnormal IP list, the request is passed; if the Pod IP in the updated value is in the abnormal IP list, the request is rejected.

[0015] In a second aspect, an embodiment of the present invention provides a device for processing abnormal endpoints of a Pod service after a node failure, comprising: The data collection module is used to collect the status data of nodes in the Kubernetes cluster after a failure and store it in the ETCD database; The data processing module is used to process the status data of the nodes in the Kubernetes cluster, filter out abnormal endpoints of Kubernetes services, and store the abnormal endpoint data of Kubernetes services in the ETCD database; The admission control module is used to perform admission control on abnormal endpoints of Kubernetes services to prevent abnormal updates and abnormal recovery.

[0016] The beneficial effects of the technical solution of the embodiment of the present invention are as follows: A method for processing abnormal endpoints of Pod services after a node failure according to the technical solution of an embodiment of the present invention comprises the following steps: step S1, collecting the status data of nodes in the Kubernetes cluster after the failure, and storing it in the ETCD database; step S2, processing the status data of nodes in the Kubernetes cluster, screening out abnormal endpoints of Kubernetes services, and storing the abnormal endpoint data of Kubernetes services in the ETCD database; step S3, performing access control on abnormal endpoints of Kubernetes services to prevent abnormal updates and abnormal recovery. The present invention can detect the abnormal status of nodes in the Kubernetes cluster, timely update the endpoint information associated with abnormal Pods on abnormal nodes, improve the fault recovery efficiency of the Kubernetes cluster, and further enhance the user experience.

[0017] The present invention realizes rapid detection and response to node anomalies by performing status monitoring and fault diagnosis of containers. In the fault scenario of node anomalies, by monitoring the Kubernetes node lease, after the lease renewal times out, the specific annotated Pod object on the node is queried, and after detecting the connectivity, the associated service backend endpoint is updated according to the result, and the abnormal endpoint update request is intercepted by the Kubernetes admission controller. Under the premise of ensuring the stability of the system, when a node failure occurs in the Kubernetes cluster, the corresponding Pod IP can be quickly removed from the Service endpoint load balancing pool, shortening the fault response time and reducing the probability of access failure, thereby improving the availability of the application and user experience.

[0018] The present invention does not involve modification of Kubernetes cluster-related parameters, nor does it perform traffic proxy or traffic hijacking. It only solves the problem of service traffic management after Kubernetes node and Pod failure through resource monitoring and operation verification methods.

[0019] A device for processing abnormal endpoints of a Pod service after a node failure in the technical solution of an embodiment of the present invention has the same beneficial effects as a method for processing abnormal endpoints of a Pod service after a node failure in the technical solution of an embodiment of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of a method for processing abnormal endpoints of a Pod service after a node failure according to an exemplary embodiment; Figure 2 It is a schematic diagram of the structure of a device for processing abnormal endpoints of a Pod service after a node failure according to an exemplary embodiment; Figure 3 is a flow chart showing a state data flow of collecting nodes in a Kubernetes cluster after a failure according to an exemplary embodiment; Figure 4 This is a flowchart of processing status data of a node in a Kubernetes cluster according to an exemplary embodiment; Figure 5 The present invention is an admission control flow chart for preventing abnormal update and abnormal recovery of abnormal endpoints of Kubernetes services according to an exemplary embodiment. DETAILED DESCRIPTION

[0021] In order to more clearly illustrate the technical features of the solution of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0022] like Figure 1 As shown, a method for processing abnormal endpoints of a Pod service after a node failure provided by an embodiment of the present invention includes the following steps: Step S1, collect the status data of the nodes in the Kubernetes cluster after the failure and store it in the ETCD database; Step S2, processing the status data of the nodes in the Kubernetes cluster, filtering out abnormal endpoints of the Kubernetes services, and storing the abnormal endpoint data of the Kubernetes services in the ETCD database; Step S3, performing admission control on abnormal endpoints of Kubernetes services to prevent abnormal updates and abnormal recovery.

[0023] As a possible implementation of this embodiment, the step S1, collecting the status data of the nodes in the Kubernetes cluster after the failure and storing it in the ETCD database, includes: Step S101, obtaining a list of node leases in the Kubernetes cluster through Kubernetes authentication; Step S102, calculating the renewal time of the node lease, and calculating the node status through the renewalTime (lease renewal time) field; Step S103, calculating the optimal abnormal waiting time of each node according to the renewal time of the node lease; Step S104, record the node status and store it in the ETCD database.

[0024] As a possible implementation of this embodiment, the step S102, calculating the renewal time of the node lease and calculating the node status through the renewalTime field, includes: Traverse all node lease lists and obtain node lease renewalTime; Calculate the difference between the node lease renewal time and the current time and convert it into milliseconds. The difference is the node lease renewal time. If the node lease renewal time is less than or equal to the optimal abnormal waiting time value of the node in memory, the node is judged to be in a normal state. If the node lease renewal time is greater than the optimal abnormal waiting time value of the node in memory, the node is judged to be in an abnormal state.

[0025] As a possible implementation of this embodiment, the step S103, calculating the optimal abnormal waiting time of each node according to the renewal time of the node lease, includes: At 00:00:00 every day in the memory, initialize an Int type one-dimensional array for each node and store the calculated node lease renewal time in the one-dimensional array; At 00:00:00 every day, the values ​​in the Int one-dimensional array of nodes of the previous day are sorted from small to large, and the 98% number of the sorted array is taken using the upward modulus method. This number is used as the optimal exception waiting time of the current node and recorded in the memory Map. The Int one-dimensional array of nodes of the previous day is released.

[0026] As a possible implementation of this embodiment, step S2, processing the status data of the nodes in the Kubernetes cluster, screening out abnormal endpoints of the Kubernetes service, and storing the abnormal endpoint data of the Kubernetes service in the ETCD database, includes: Step S201, traverse the node status list, and if there is an abnormal node, obtain the Pod list with the specified annotation on the abnormal node; Step S202, traverse the Pod list to detect whether the Pod IP is reachable. If the Pod IP is not reachable, record the Pod in the abnormal Pod list; Step S203, traverse the IP unreachable Pod list generated in the previous step, and obtain the corresponding Service list according to the label; Step S204, traverse the generated Service list and delete abnormal Pod IP information in the endpoint; Step S205, record abnormal node information, abnormal endpoint information and abnormal Pod IP information, and store them in the ETCD database.

[0027] As a possible implementation of this embodiment, step S3, performing admission control on abnormal endpoints of Kubernetes services to prevent abnormal updates and abnormal recovery thereof, includes: Step S301, using the Kubernetes admission control ValidatingAdmissionWebhook application to monitor the update of the endpoint; Step S302: judging whether the endpoint of the arrival event has an associated fault node according to the abnormal endpoint information recorded in the ETCD database; Step S303, obtaining the current state of the associated faulty node according to the node state information recorded in the ETCD database, if the current state of the node is normal, deleting the abnormal endpoint information in the ETCD database; if the current state of the node is abnormal, executing the next step; Step S304: According to the faulty Pod IP list recorded in the ETCD database, determine whether the IP updated by the current endpoint update method is a faulty IP. If it is a faulty Pod IP, reject the request.

[0028] As a possible implementation of this embodiment, step S302 includes: Use the Namespace+Name of the endpoint that triggers the update event as the key value to query the endpoint list / registry / node-detect / endpoint in the ETCD database. If the key value does not exist, the process ends. If the key value exists, get the corresponding value and split the value according to the character '-' to get a values ​​array. The first element of the values ​​array is the node name, and the second element is the abnormal Pod IP list; Use the node name as the key value to query the node status in the node status list / registry / node-detect / node-status of the etcd database.

[0029] As a possible implementation of this embodiment, step S303 includes: If the obtained associated node status is normal, delete the record corresponding to the endpoint Namespace+Name as the key value in the endpoint list / registry / node-detect / endpoint, and the process ends; If the status of the associated node obtained in the step is abnormal, get the second element of the values ​​array, which is the abnormal PodIP list string.

[0030] As a possible implementation of this embodiment, step S304 includes: Use the character ';' to cut the abnormal Pod IP list string to get the abnormal IP list; Get the Pod IP value corresponding to the current endpoint resource update operation. If the Pod IP in the updated value is not in the abnormal IP list, the request is passed; if the Pod IP in the updated value is in the abnormal IP list, the request is rejected.

[0031] In a second aspect, an embodiment of the present invention provides a device for processing abnormal endpoints of a Pod service after a node failure, comprising: The data collection module is used to collect the status data of nodes in the Kubernetes cluster after a failure and store it in the ETCD database; The data processing module is used to process the status data of the nodes in the Kubernetes cluster, filter out abnormal endpoints of Kubernetes services, and store the abnormal endpoint data of Kubernetes services in the ETCD database; The admission control module is used to perform admission control on abnormal endpoints of Kubernetes services to prevent abnormal updates and abnormal recovery.

[0032] The present invention first obtains the Kubernetes node status, discovers the abnormal status of the node in the Kubernetes cluster, and promptly updates the endpoint information associated with the abnormal Pod on the abnormal node, thereby improving the fault recovery efficiency of the Kubernetes cluster and further improving the user experience.

[0033] like Figure 3 As shown, the specific process of collecting the status data of the nodes in the Kubernetes cluster after the failure of the present invention is as follows.

[0034] Step S101, obtaining a list of node leases in the Kubernetes cluster.

[0035] Lease is a resource type in Kubernetes, and its GVK is coordination.k8s.io / v1 / Lease. The present invention obtains the Node Lease resource list under the kube-node-lease namespace through Kubernetes authentication.

[0036] Step S102, calculate the node lease renewal time and determine the node status.

[0037] kube-node-lease contains the Lease objects associated with each node. Node leases allow node-related components in Kubernetes to send heartbeats and update the renewTime field of the Node Lease resource.

[0038] The present invention traverses all Node Lease resources regularly (every 1 second) to obtain Lease renewTime. The node status is calculated through the renewTime field. The specific method is as follows: The difference between renewTime and the current time is converted into milliseconds, which is the node lease renewal time. If the node lease renewal time is less than or equal to the optimal abnormal waiting time value of the node in the memory, the node is judged to be in a normal state. If the node lease renewal time is greater than the optimal abnormal waiting time value of the node in the memory, the node is judged to be in an abnormal state, and the node status list is recorded in the memory list. The initial value of the optimal abnormal waiting time of the node is 15000 (milliseconds), which is subsequently calculated and updated through step 103.

[0039] Step S103, calculating the optimal exception waiting time of each node according to the lease renewal time.

[0040] In step S103 of the present invention, the optimal abnormal waiting time can be calculated for each node respectively, and the calculation scheme is as follows: (1) At 00:00:00 every day in the memory, initialize an Int type one-dimensional array for each node, and store the node lease renewal time calculated in step S102 into the one-dimensional array; (2) At 00:00:00 every day, sort the values ​​in the Int one-dimensional array of nodes from the previous day from small to large, and take the 98th (modulo upward) number of the array. This number is used as the optimal exception waiting time for the current node, recorded in the memory map, and the Int one-dimensional array of nodes from the previous day is released.

[0041] Step S104, record the node status into the ETCD database, and execute the node status processing flow.

[0042] Insert or update the ETCD database table / registry / node-detect / node-status, where the key is the node name and the value is 0 or 1, where 0 is normal and 1 is abnormal. Figure 2 The process of handling service exception endpoints under abnormal conditions.

[0043] like Figure 4 As shown, the specific process of the present invention for processing the status data of the nodes in the Kubernetes cluster is as follows.

[0044] Step S201, traverse the node status list, if there is an abnormal state node, obtain the Pod list with the specified annotation on the abnormal node.

[0045] Traverse the abnormal node list. Get the list of all Pods with node-detect:true annotation under the abnormal node according to the node name.

[0046] Step S202, traverse the Pod list generated in the previous step, check whether the Pod IP is reachable, and record the Pod list whose IP is unreachable.

[0047] Traverse the Pod list in step S201 to check whether the corresponding Pod's IP is reachable, with a timeout of 1000 milliseconds. If the Pod IP is unreachable, record the Pod in the abnormal Pod list.

[0048] Step S203, traverse the IP unreachable Pod list generated in the previous step, and obtain the corresponding Service list according to the label.

[0049] Traverse the Pod list in step S202, and obtain the associated Service resource list by querying the matchLabels of the abnormal Pod.

[0050] Step S204, traverse the Service list generated in the previous step, and delete the abnormal PodIP information in the backend endpoints.

[0051] Traverse the Service list in step S203, obtain the corresponding Endpoint endpoint resource through Service labels, update the relevant Endpoint endpoint information, and delete the record corresponding to the abnormal Pod IP in the subnets.addresses of the endpoint.

[0052] Step S205, record abnormal node information, abnormal endpoint information, and abnormal Pod IP information into the ETCD database.

[0053] Insert or update endpoint information in the ETCD database table / registry / node-detect / Endpoint. The specific operations are: The combination of the Namespace and Name of the Endpoint endpoint is used as the key, in the format of Endpoint Namespace+"-"+Endpoint Name. The value is the abnormal node information deleted from the endpoint, in the format of nodeName+"-"+PodIP. If there are multiple abnormal Pods under the same endpoint, multiple records are separated by ";". For example, for the record with the key test-nginx, the value is node02-10.245.241.78;10.245.241.79;10.245.241.88.

[0054] Figure 3 and Figure 4 The middle process implements node anomaly detection and detection and processing of abnormal endpoints of Kubernetes Service. Figure 5 The present invention performs an admission control process for preventing abnormal updates and abnormal recovery of abnormal endpoints of Kubernetes services. The process can prevent abnormal Pod IP information that has been deleted from the abnormal endpoint from being mistakenly added back to the abnormal endpoint.

[0055] like Figure 5 As shown, the specific process of the present invention for performing admission control on abnormal endpoints of Kubernetes services to prevent abnormal updates and abnormal recovery is as follows.

[0056] Step S301, use Kubernetes admission control ValidatingAdmissionWebhook to monitor the update method of resource Endpoints.

[0057] The admission control application is deployed in the cluster as a Kubernetes ValidatingAdmissionWebhook application to detect the update method of the Endpoint endpoint resource.

[0058] Step S302: According to the abnormal Endpoint information recorded in the ETCD database, it is determined whether the Endpoint of the arrival event has an associated fault node.

[0059] Use the Namespace+Name of the Endpoint that triggers the update event as the key value to query the ETCD database table / registry / node-detect / endpoint. If the key value does not exist in the table, the process ends. If the key value exists in the table, get the corresponding value. Split the value according to the character '-' to get a values ​​array, where the first element of the values ​​array is the node name and the second element is the abnormal Pod IP list. Use the node name as the key value to query the node status in the ETCD database table / registry / node-detect / node-status.

[0060] Step S303, obtain the current state of the faulty node associated in the previous step according to the node state information recorded in the ETCD database. If the current state of the node is normal, delete the abnormal Endpoint information in the ETCD database; if the current state of the node is abnormal, execute the next step.

[0061] If the status of the associated node obtained in step S302 is normal, the process ends after deleting the record corresponding to the Namespace+Name of the Endpoint as the key value in the / registry / node-detect / endpoint table; If the associated node status obtained in step S302 is abnormal, obtain the second element of the values ​​array in step S302, which is the abnormal Pod IP list string.

[0062] Step S304: According to the list of faulty Pod IPs recorded in the ETCD database, determine whether the IP updated by the current Endpoint update method is a faulty IP. If it is a faulty Pod IP, reject the request.

[0063] The abnormal Pod IP list string in step S303 is cut by the character ';' to obtain the abnormal IP list. Get the Pod IP value corresponding to the current Endpoint resource update operation. If the Pod IP in the updated value is not in the abnormal IP list, the request is passed; if the Pod IP in the updated value is in the abnormal IP list, the request is rejected.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific implementation modes of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for processing abnormal endpoints of Pod services after node failure, characterized in that: The steps include: Step S1, collect the status data of the nodes in the Kubernetes cluster after the failure and store it in the ETCD database; Step S2, processing the status data of the nodes in the Kubernetes cluster, filtering out abnormal endpoints of the Kubernetes services, and storing the abnormal endpoint data of the Kubernetes services in the ETCD database; Step S3, performing admission control on abnormal endpoints of Kubernetes services to prevent abnormal updates and abnormal recovery.

2. The method for processing abnormal endpoints of Pod services after node failure according to claim 1 is characterized in that: The step S1 collects the status data of the nodes in the Kubernetes cluster after the failure and stores it in the ETCD database, including: Step S101, obtaining a node lease list in the Kubernetes cluster through Kubernetes authentication; Step S102, calculate the renewal time of the node lease, and calculate the node status through the renewalTime field; Step S103, calculating the optimal abnormal waiting time of each node according to the renewal time of the node lease; Step S104, record the node status and store it in the ETCD database.

3. The method for processing abnormal endpoints of Pod services after node failure according to claim 2 is characterized in that: The step S102, calculating the renewal time of the node lease, and calculating the node status through the renewalTime field, includes: Traverse all node lease lists and obtain node lease renewalTime; Calculate the difference between the node lease renewal time and the current time and convert it into milliseconds. The difference is the node lease renewal time. If the node lease renewal time is less than or equal to the optimal abnormal waiting time value of the node in memory, the node is judged to be in a normal state. If the node lease renewal time is greater than the optimal abnormal waiting time value of the node in memory, the node is judged to be in an abnormal state.

4. The method for processing abnormal endpoints of Pod services after node failure according to claim 2 is characterized in that: The step S103, calculating the optimal abnormal waiting time of each node according to the renewal time of the node lease, includes: At 00:00:00 every day in the memory, initialize an Int type one-dimensional array for each node and store the calculated node lease renewal time in the one-dimensional array; At 00:00:00 every day, the values ​​in the Int one-dimensional array of nodes of the previous day are sorted from small to large, and the 98% number of the sorted array is taken using the upward modulus method. This number is used as the optimal exception waiting time of the current node and recorded in the memory Map. The Int one-dimensional array of nodes of the previous day is released.

5. The method for processing abnormal endpoints of Pod services after node failure according to any one of claims 1 to 4, characterized in that: The step S2 processes the status data of the nodes in the Kubernetes cluster, filters out abnormal endpoints of the Kubernetes services, and stores the abnormal endpoint data of the Kubernetes services in the ETCD database, including: Step S201, traverse the node status list, and if there is an abnormal node, obtain the Pod list with the specified annotation on the abnormal node; Step S202, traverse the Pod list to detect whether the Pod IP is reachable. If the Pod IP is not reachable, record the Pod in the abnormal Pod list; Step S203, traverse the IP unreachable Pod list generated in the previous step, and obtain the corresponding Service list according to the label; Step S204, traverse the generated Service list and delete abnormal Pod IP information in the endpoint; Step S205, record abnormal node information, abnormal endpoint information and abnormal Pod IP information, and store them in the ETCD database.

6. The method for processing abnormal endpoints of Pod services after node failure according to claim 5 is characterized in that: The step S3 performs admission control on abnormal endpoints of the Kubernetes service to prevent abnormal updates and abnormal recovery, including: Step S301, using the Kubernetes admission control ValidatingAdmissionWebhook application to monitor the update of the endpoint; Step S302: judging whether the endpoint of the arrival event has an associated fault node according to the abnormal endpoint information recorded in the ETCD database; Step S303, obtaining the current state of the associated faulty node according to the node state information recorded in the ETCD database, if the current state of the node is normal, deleting the abnormal endpoint information in the ETCD database; if the current state of the node is abnormal, executing the next step; Step S304: According to the faulty Pod IP list recorded in the ETCD database, determine whether the IP updated by the current endpoint update method is a faulty IP. If it is a faulty Pod IP, reject the request.

7. The method for processing abnormal endpoints of Pod services after node failure according to claim 6 is characterized in that: The step S302 includes: Use the Namespace+Name of the endpoint that triggers the update event as the key value to query the endpoint list of the ETCD database. If the key value does not exist, the process ends. If the key value exists, get the corresponding value and split the value according to the character '-' to get a values ​​array. The first element of the values ​​array is the node name, and the second element is the abnormal Pod IP list; Use the node name as the key value to query the node status in the node status table of the etcd database.

8. The method for processing abnormal endpoints of Pod services after node failure according to claim 7 is characterized in that: The step S303 includes: If the obtained associated node status is normal, delete the record corresponding to the endpoint's Namespace+Name as the key value in the endpoint list, and the process ends; If the status of the associated node obtained in the step is abnormal, get the second element of the values ​​array, which is the abnormal Pod IP list string.

9. The method for processing abnormal Pod service endpoints after node failure according to claim 8, characterized in that: The step S304 includes: Use the character ';' to cut the abnormal Pod IP list string to get the abnormal IP list; Get the Pod IP value corresponding to the current endpoint resource update operation. If the Pod IP in the updated value is not in the abnormal IP list, the request is passed; if the Pod IP in the updated value is in the abnormal IP list, the request is rejected.

10. A device for processing abnormal endpoints of Pod services after a node failure, characterized in that: include: The data collection module is used to collect the status data of nodes in the Kubernetes cluster after a failure and store it in the ETCD database; The data processing module is used to process the status data of the nodes in the Kubernetes cluster, filter out abnormal endpoints of Kubernetes services, and store the abnormal endpoint data of Kubernetes services in the ETCD database; The admission control module is used to perform admission control on abnormal endpoints of Kubernetes services to prevent abnormal updates and abnormal recovery.

Images