Container group health scoring method based on kubernetes

Abstract

The invention discloses a kubernetes-based container group health scoring method, and belongs to the technical field of computers. The method comprises the following steps of: creating a HealthCheckPolicy health scoring rule; the health scoring rule of the HealthCheckPolicy comprises a lifecycle type and an alert type, and the health scoring rule of the HealthCheckPolicy comprises a lifecycle type and an alert type; obtaining alarm information and pod information; according to the alarm information, the pod information and a HealthCheckPolicy health scoring rule, a health score is obtained through calculation; and restarting the container according to the health score. According to the method, the health condition of the running application can be evaluated, and the application in the sub-health state can be restarted gracefully.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a container group health scoring method based on Kubernetes. Background Technology

[0002] After an application has been running for a period of time, unexpected behaviors may occur, such as memory leaks. This largely depends on the application code, that is, whether the developers consciously avoid these problems. We cannot guarantee that all applications will not encounter problems after running for a long time; therefore, it is necessary to perform health scoring on applications based on their running time to assess their health.

[0003] In addition to assessing the health of an application based on its runtime, alerts issued by the application itself are also important reference indicators. Applications expose monitoring metrics and customize alert rules according to their own circumstances. Therefore, when an application experiences certain foreseeable problems, the values ​​of the monitoring metrics will trigger the alert rules. We can then further assess the health of the application based on the alerts and the alert penalty rules.

[0004] When an application is in a sub-healthy state, it needs to be gracefully restarted within a specified time window and without affecting the business operations. Summary of the Invention

[0005] The purpose of this invention is to provide an efficient method for health scoring of Kubernetes-based container groups.

[0006] To address the aforementioned technical problems, this invention provides a container group health scoring method based on Kubernetes, comprising the following steps: Create HealthCheckPolicy health scoring rules; Get alarm information and pod information; The health score is calculated based on alarm information, pod information, and HealthCheckPolicy health scoring rules; Restart the container based on the health score.

[0007] Preferably, a health score is calculated based on alarm information, pod information, and HealthCheckPolicy health scoring rules, specifically including the following steps: The final deduction for the lifecycle type is calculated based on the expected container lifetime retention of the pod information and the sub-health value in the HealthCheckPolicy health scoring rules. The final deduction for the alert type is calculated based on the alert information and the alert deduction details in the HealthCheckPolicy health scoring rules. A health score is calculated based on the final deductions for the lifecycle type and the alert type.

[0008] Preferably, the final deduction for the lifecycle type is calculated based on the expected container lifetime retention of the pod information and the sub-health value in the HealthCheckPolicy health scoring rules, specifically including the following steps: Create a timer; When the timer reaches the preset time, retrieve all managed Pods; The process involves iterating through the Pods and deducting points for each Pod based on its sub-health value and expected container lifespan retention, resulting in a final deduction for the lifecycle type.

[0009] Preferably, a score is deducted for each Pod based on its sub-health value and expected container lifespan retention, resulting in a final lifecycle-type score. This process includes the following steps: To determine if each Pod meets the health score criteria: compare the container startup time with the initial delay time before container startup. If the container startup time is lower than the initial delay time before container startup, then it does not meet the health score criteria. Also, determine the current container status. If the container is in a Terminating or non-Running state, then it does not meet the health score criteria. When a Pod has a health score, the time interval for deducting points is calculated as expected container lifespan / [health value - sub-health value] (periodTime). When the Pod information has a transition time (lastTransitionTime), the deduction value is calculated as [current time - lastTransitionTime] / time interval (periodTime). When the Pod information does not have a transition time (lastTransitionTime), the deduction value is calculated as [current time - container startup time] / time interval (periodTime). Based on this deduction value, the final deduction for the lifecycle type is calculated.

[0010] Preferably, the calculation process for the final deduction of the lifecycle type is as follows: The time interval for deducting points is periodTime: In the formula: S0 represents the time interval for point deduction (periodTime); S0 represents the initial health value; H sub Represents the threshold of sub-health; The points deducted this time are: In the formula: This represents the deduction value for this instance; t represents the current time. Represents the transition time, lastTransitionTime; The final deduction for the lifecycle type is: ; In the formula: The final deduction is for the lifecycle type; w l Represents lifecycle weight.

[0011] Preferably, the final deduction for the alert type is calculated based on the alert information and the alert deduction details in the HealthCheckPolicy health scoring rules, specifically including the following steps: Based on the alarm information, the corresponding pod and its namespace can be parsed out; Retrieve the alarm deduction details from the HealthCheckPolicy health scoring rules corresponding to this pod; Based on the alarm deduction details and alarm information, the corresponding deductions are applied to the pods, and the final deduction for the alarm type is calculated.

[0012] Preferably, based on the alarm deduction details and alarm information, corresponding deductions are applied to the pod to calculate the final deduction for the alert type. The calculation formula is as follows: Let alarm a be an alarm set A containing alarm information. j The number of times it appears is denoted as c. j Map the single deduction p according to the alarm level j The deduction for a normal alarm is: To avoid significant point deductions caused by alarm storms, an EMA smoothing strategy is introduced: Define the time constant T j Smoothing factor: Then the smoothing state a of the alarm intensity j (t) is: in The final deduction for alert type is: If an alarm has healed itself, initiate an exponential decay recovery strategy to allow for slow recovery: Different λ values ​​are configured for different alarm levels. λ can be obtained using the following formula, where r is the decay rate per minute: In the formula: Represents the time interval between the current time and the initial time; : This refers to the total points deducted for alert types. This refers to the point deduction for a specific alarm in alarm set A. This refers to a warning that has self-healed. The deductions calculated using the exponential decay recovery algorithm after a time interval; This refers to the points deducted for each specified scoring strategy within the policy types; Sum the deductions for each strategy. λ refers to the decay rate over the corresponding continuous time period; w a Refers to the alert weight.

[0013] Preferably, the formula for calculating the health score is: In the formula: Refers to health score; This refers to summing the deductions for each strategy. The final deduction for lifycycle and the final deduction for alert type are added together.

[0014] Preferably, the container is restarted based on a health score, specifically including the following steps: The health status of a Pod is determined based on the latest health score. A Pod is considered healthy when its health score is above the sub-health threshold; a Pod is considered sub-healthy when its health score is below the sub-health threshold but above the unhealthy threshold; and a Pod is considered unhealthy when its health score is below the unhealthy threshold. When the Pod is in a healthy state, it does not need to be restarted; When a Pod is in a sub-healthy state, check if the time is within the period configured; whether the number of ongoing restart requests in the namespace to which the current Pod belongs has reached the limit; and whether the number of ongoing restart requests in the load to which the current Pod belongs has reached the limit. Restart the container when the Pod is in an unhealthy state.

[0015] Preferably, restarting the container includes the following steps: Listen for containerRecreateRequests belonging to this node; Determine if the currently running restart tasks have reached their limit; Determine if there are any containers in the same pod that are already restarting; Gain graceful restart tolerance time; The execPreStopHook executes the operations performed before the application stops. The remaining graceful start / stop time is determined based on the time elapsed after execPreStopHook; if the remaining graceful start / stop time is less than 30 seconds, it is reset to 30 seconds. The StopContainer command is used to delete a container. Write the restart results back to crrstatus. If there is a forced stop during the restart process, set isGrace to false.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention can accurately assess the health status of running applications and gracefully restart applications in a sub-healthy state. Attached Figure Description

[0017] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0018] Figure 1 This is a schematic diagram of the overall architecture design; Figure 2 This is a diagram illustrating the scoring and restart process; Figure 3 This is a schematic diagram of a scoring process based on runtime. Figure 4 This is a schematic diagram of the scoring process based on alarm rules; Figure 5 This is a flowchart illustrating the restart assessment module process; Figure 6 This is a schematic diagram of the container restart module process. Detailed Implementation

[0019] Numerous specific details are set forth in the following description to provide a full understanding of the invention. However, the invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0020] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0021] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0022] The present invention will now be described in further detail with reference to the accompanying drawings: To better illustrate the technical effects of the present invention, the present invention provides the following specific embodiments to illustrate the above technical process: Example 1: A Kubernetes-based container group health scoring system that can assess the health status of running applications and gracefully restart applications in a sub-healthy state.

[0023] To address the aforementioned technical problems, this invention provides a container group health scoring system based on Kubernetes.

[0024] The system relies on three custom resources: HealthCheckPolicy, PodHealthScore, and ContainerRecreateRequest. These three resources are interdependent and work together. HealthCheckPolicy, or health scoring policy, defines the health scoring strategy for a group of Pods, including Pod label selection, Pod container restart policy, and Pod health scoring method. It consists of three main parts: 1) scope: Indicates the area of ​​effect of the policy. 2) restartPolicy: A rule that instructs the pod to be restarted. 2.1) subhealthyThreshold: Specifies the subhealth threshold. When the pod score is less than or equal to subhealthyThreshold and greater than unhealthyThreshold, it must be restarted within the time period specified by period. 2.2) unhealthyThreshold: Specifies the unhealthy threshold. When the pod score is less than or equal to this value, the pod must be restarted immediately. 2.3) enable: Indicates whether the restart rule is enabled. 2.4) constraints: Indicates certain restrictions when performing a restart operation. 2.4.1) maxConcurrentTotal: Indicates the maximum number of tasks that can be restarted simultaneously across the entire cluster. 2.4.2) maxConcurrentPerNamespace: Indicates the maximum number of tasks that can be restarted simultaneously in each namespace during a restart. 2.4.3) maxConcurrentPerNode: Indicates the maximum number of tasks that can be executed concurrently in each node. 2.4.4) minimalAlivePerWorkload: Minimum number or percentage of computers to keep alive for each workload. 2.4.5) restartWindow: A time window during which the application can be restarted when it is in a sub-healthy state. It is the start and end time defined by the cron expression. 3) healthCheckPolicies: Rules that indicate health scores 3.1) periodSeconds: Health check intervals 3.2) Policies: The policies are configured with factors that affect the health score, and each checkpoint is responsible for deducting points from the health value.

[0025] 3.2.1) Lifecycle Type: This type calculates the health deduction cycle based on the sub-health value and the container's set lifespan. The specific algorithm will be explained in detail below.

[0026] 3.2.2) Alert Type: This type configures various alarm levels and corresponding penalty rules, and also allows for individual configuration of specific alarm penalty rules. When an alarm is received, points are deducted from the application according to the penalty rules. The specific algorithm will be explained in detail below.

[0027] The lifecycle and alert types are rules configured under policies. You need to specify the type as lifecycle or alert. The program will check the type and execute the corresponding check logic according to the lifecycle or alert type to perform subsequent scoring.

[0028] PodHealthScore, or Pod Health Score, is used to record the score details of a Pod in a certain health scoring strategy. It mainly records the current health status of each container, including healthy / sub-healthy status, current health score, number of restarts, last restart time, and last score update time.

[0029] The ContainerRestartRequest, or container restart request, instructs how the container should be restarted, specifically including: Basic information about the container to be restarted includes: ailurePolicy: 'Fail' or 'Ignore', indicating that CRR will end immediately once a container stops or fails to rebuild; orderedRecreate: 'true' means that the rebuild will only begin after the previous container has been completely rebuilt. terminationGracePeriodSeconds: The time to wait for the container to gracefully exit. unreadyGracePeriodSeconds: Sets the Pod to not ready before rebuilding and waits for this period before starting the rebuild. minStartedSeconds: The minimum amount of time the new container must remain running after rebuilding before the container is considered successfully rebuilt. activeDeadlineSeconds: If a CRR execution exceeds this time, it is marked as terminated (containers that have not terminated are marked as failed). completionTime: CRR end time isKilled: Whether the container is terminated. isGraceful: Whether to perform a graceful restart Phase: Recreating / Failed / Succeeded / Pending, the current execution phase. message: necessary information Preferably, the health-manager is divided into two modules: a health scoring module and a restart module; Users build application health check rules by creating HealthCheckPolicy. The health scoring module first finds matching pods and injects readinessGate information into the pods so that the business is not affected during graceful restart. It then reads the health scoring rules in the HealthCheckPolicy resource and scores the health status of the specified pods based on their content. The scoring criteria include the application's runtime and the application's alarm status. The scoring results will be stored in the PodHealthScore resource, and users can view the health score of the Pod through the PodHealthScore resource; The restart module creates a new ContainerRecreateRequest resource based on the restart policy and by reading the results from PodHealthScore. The health-deamon then restarts the corresponding container according to the ContainerRecreateRequest configuration, updating the process and results in the ContainerRecreateRequest. If the restart fails, a restart exception alert is issued. The overall architecture diagram is as follows: Figure 1 As shown.

[0030] Preferably, the workflow of the health scoring and restart module is as follows: Figure 2 As shown: 1. Users create HealthCheckPolicy health scoring rules; 2. Both the health scoring module and the restart module will obtain the specific configuration of HealthCheckPolicy; 3. The health scoring module receives application alerts as a scoring basis. These alerts are typically exposed by the alertmanager component. At the same time, it queries the application's pod information in the cluster in real time and performs health assessments based on runtime. 4. The health score module updates the pod health score and stores the result in PodHealthScore; 5. Restart the module to obtain PodHealthScore information; 6. Restart the module to create a ContainerRecreateRequest container to rebuild the request; 7. health-daemon, which is a daemon process running on each node, obtains ContainerRecreateRequest information; 8. If the container corresponding to this ContainerRecreateRequest is on this node, then restart the container; 9. After restarting, return the restart result and record it in ContainerRecreateRequest; 10. Restart the module to obtain the restart result; 11. If an abnormality occurs during restart, an alarm will be issued promptly.

[0031] For the health scoring algorithm, the following notation conventions are defined: 1) The health value S(t) of the container ∈ [0,100], and the initial health value S0=100; 2) Transition Time: lastTransitionTime: t start Current time: t; 3) Sub-health threshold: H sub ; 4) Expected container lifetime retention: L (seconds, e.g., 2 days = 172800s); 5) Alarm set: A = {a j}; 6) Weight: Lifecycle weight w l alert weight w a This value is used to adjust the relative importance; the default value is 0.5. For the lifecycle type, the time interval periodTime required to deduct one point is: In the formula: This represents the time interval (periodTime) required to deduct one point. The points deducted this time are: The final deduction for the lifecycle type is: For the alert type, each alert a in alert set A j The number of times it appears is denoted as c. j Points deduction per instance are mapped to severity level. j For example, sev 1->1 point, sev2->3 points, sev3->7 points, the normal deduction for a certain alarm is: To avoid a significant penalty due to an alarm storm (i.e., a large number of alarms in a short period of time), an EMA smoothing strategy is introduced: Define the time constant T j (Seconds represent the smoothing duration), smoothing factor: Then the smoothing state a of the alarm intensity j (t) is: in For example, alarm a j The severity level is sev2, and the value is T. j =180s, at minute 0, a j (0) = 0, a continuous alarm occurs in the first minute, α ≈ 0.29, a j (1) = 0.29, the penalty is 0.87 points; in the 2nd minute α ≈ 0.49, a j (2) = 0.64, the penalty is 1.92 points; in the 3rd minute α ≈ 0.63, a j (3) = 0.88, the penalty is 2.64 points; in the 4th minute, α ≈ 0.73, a j (4) = 0.97, which is close to 1. The alarm is stabilizing and the penalty is 2.91 points.

[0032] The final deduction for alert type alerts is: Additionally, if an alarm has self-healed, an exponential decay recovery strategy is activated to allow for slow recovery: Different λ values ​​are configured for different alarm levels. λ refers to the decay rate over a continuous time period (per second). Higher alarm levels have longer recovery periods, while lower alarm levels can recover faster. λ can be obtained using the following formula, where r is the decay rate per minute. For example, for a sev 1 level alarm, if we want to recover about half the score after half an hour, then r can be taken as 2%. λ can be calculated to be approximately 0.00036. After 30 minutes, the penalty score for this alarm will decrease to approximately 52%. The final health score is: In the formula: Represents the time interval between the current time and the initial time; : This refers to the total points deducted for alert types. This refers to the deduction of a certain alarm in alarm set A, according to the exponential decay recovery formula. In the middle, the alarm self-heals at time t, but by time t2 for calculating the score, time t2 has passed. =t2-t, then It means that a certain warning is self-healing. The deductions calculated using the exponential decay recovery algorithm after a time interval; This refers to the deduction of points for each specified scoring strategy in the policies, namely the lifecycle type and the alert type. Summing the deductions for each strategy, then... That is the final score.

[0033] This refers to the deduction for each specified scoring strategy in the policies, i.e., the lifycycle type deduction in the formula above. Deduction of points for alert type ; Preferably, for the lifecycle type of the health scoring module, the specific algorithm is as follows: Figure 3 As shown: The LifeCheck health scoring module listens to the HealthCheckPolicy resource and manages HealthCheckWorkers based on that resource. These Workers are responsible for scoring and recording the scores.

[0034] The Worker process mainly involves the following steps: 1. Create a timer based on spec.healthCheckPolicies.periodSeconds. The next score will be triggered after the timer has elapsed for the specified period of .periodSeconds. 2. After entering the scoring logic, it first retrieves all managed Pods based on spec.podSelector; 3. Iterate through the Pods, and perform a health score for each Pod here; a. Determine if the Pod meets the health score criteria; i. Compare the current time - container startup time with the initial delay time before container startup (spec.initialDelaySeconds). If the container startup time is lower than the initial delay time before container startup, no scoring is required because the service may not be available at this time. ii. Determine the current container status. If the container is in a Terminating or non-Running state, it means that the container may have been created normally or that the container is being intervened by the health scoring system (restarting because the score is too low). b. Calculate the time interval (periodTime) required to deduct 1 point based on the expected container lifespan (retention) of the lifecycle type / [health value - sub-health value]; The health value is fixed at 100 (i.e., S0=100), and the subhealth value is the subhealthyThreshold parameter configured in the HealthCheckPolicy health scoring rules. If it is configured to be 60, the health value minus the subhealth value is 100-60. c. Check if the HealthCheckPolicy status contains a transition time lastTransitionTime (lastTransitionTime is updated to the current value after each check is completed); i. If it exists, calculate the deduction value for this time by [current time - lastTransitionTime] / time interval periodTime; ii. If it does not exist, the deduction value for this time is calculated by [current time - container startup time] / time interval periodTime (the transition time lastTransitionTime has not yet been recorded in the first calculation); d. When the current deduction value is greater than 0, update the score field in the PodHealthScore. 4. After the traversal is complete, wait for the timer to notify you of the next health score.

[0035] Preferably, for the alert type of the health scoring module, the specific algorithm is as follows: Figure 4 As shown: 1. When the alertmanager sends an alert message to the webhook interface of the health scoring component, the alertCheck module will be triggered; 2. The alertCheck module will parse the corresponding pod and its namespace based on the alert information; 3. The alertCheck module reads the healthCheckPolicy and obtains the alarm deduction details corresponding to the pod; 4. Deduct points from the pods according to the details of the deductions and the level of the alarm information received.

[0036] Preferably, for the restart evaluation module, the specific algorithm is as follows: Figure 5 As shown: The controller determines the health status of a Pod based on its latest PodHealthScore. When the healthCheck score (health score) is higher than the sub-health threshold, the Pod is considered healthy and does not need to be restarted. When the healthyCheck score is below the sub-health threshold but above the unhealthy threshold, the Pod is considered to be in a sub-healthy state, and will then be restarted if the following three conditions are met: Check if the time is within the time range configured in the period configuration; Has the number of ongoing restart requests for the namespace to which the current pod belongs reached its limit? Has the current pod's ongoing restart requests reached their limit?

[0037] When the healthyCheck score is lower than the unhealthy threshold, the Pod is considered unhealthy, and the controller will immediately create a restart request. Before creating a restart request, you need to set the readiness gate healthycheck in the pod to false.

[0038] Preferably, the pod's readinessGate module also needs to be set during initialization, and the specific process is as follows: When restarting the evaluation module, when creating a new pod, the `app.harmonylcoud.cn / healthyCheck` check item needs to be injected into the pod's `readinessGate` in the mutation webhook, and the `app.harmonylcoud.cn / healthyCheck` item in the corresponding status should be set to `true`.

[0039] Before creating a new container restart request, the restart assessment module needs to set the corresponding app.harmonylcoud.cn / healthyCheck to false to ensure that the pod IP is kicked out of the service.

[0040] After the container restarts, restarting the evaluation module requires setting app.harmonylcoud.cn / healthyCheck to true again.

[0041] Preferably, the detailed process for the container restart module is as follows: Figure 6 As shown: 1. Listen for containerRecreateRequests belonging to this node; 2. Determine if the currently running restart tasks have reached their limit; 3. Determine if there are any containers in the same pod that are already restarting (the orderedRecreate property in CRR should always be true); 4. Entering KillContainer logic: a. Obtain the gracePeriod based on the configuration, which is the grace restart tolerance time. gracePeriod can be specified through TerminationGracePeriodSeconds in the pod's YAML. The default value of this parameter is 30 seconds. The application needs to configure this parameter according to its own grace restart time.

[0042] b.execPreStopHook executes the operations performed before the application stops. i. Execute the program configured at the PreStop hook point using HookRunner.Run, receive the results via a Channel, and return the time elapsed during this process. c. Calculate the remaining graceful start / stop time (total time - time consumed by PreStop) based on the time returned after execPreStopHook. d. If the remaining time is less than 30 seconds, it will be reset to 30 seconds. e.StopContainer performs the operation of deleting a container. i.runtimeClient.StopContainer. Calls the CRI interface to execute the container's Stop method.

[0043] 1. Send a SIGTERM signal to the Container 2. Construct a timer, where timeout is the remaining graceful restart tolerance time. 3. If the timer expires and the container has not been deleted, send a SIGKILL signal to the container. 4. The container is deleted, waiting for kubelet to call docker to rebuild it. f. Write the restart results back to crrstatus. i. If there is a forced stop during the restart process, set isGrace to false, which contains detailed timelines.

[0044] Preferably, the system also exposes a variety of monitoring indicators to monitor the operating status of components in real time and to issue timely alerts when anomalies occur. The specific monitoring indicators are as follows: The total number of tasks currently restarting in the cluster. The total number of tasks currently queued in the cluster, waiting to be restarted. Total number of tasks running in the current cluster Number of tasks currently being restarted on a given node The number of tasks currently queued and waiting to be restarted in a certain node. Total number of tasks running in a certain node Number of tasks currently undergoing restart in a certain namespace The number of tasks currently queued in a certain namespace, waiting to be restarted. Total number of tasks run in a certain namespace Number of times a container in a certain node fails to restart Ungraceful restart count of a container in a certain node Number of successful restarts of a certain container in a certain node The duration of prehook execution for a container in a certain node before its most recent restart The duration of the kill operation performed on a container in a node before its most recent restart. The time taken for a container in a certain node to start successfully after its most recent restart. In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules, units, or units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units, modules, or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0045] The units may or may not be physically separate. The components shown as units can be one or more physical units, meaning they can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0046] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0047] In particular, according to embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined in the methods of this invention. It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0048] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0049] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions within the technical scope disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A container group health scoring method based on Kubernetes, characterized in that, Includes the following steps: Create HealthCheckPolicy health scoring rules; Get alarm information and pod information; The health score is calculated based on alarm information, pod information, and HealthCheckPolicy health scoring rules; Restart the container based on the health score.

2. The Kubernetes-based container group health scoring method according to claim 1, characterized in that, The health score is calculated based on alarm information, pod information, and HealthCheckPolicy health scoring rules, specifically including the following steps: The final deduction for the lifecycle type is calculated based on the expected container lifetime retention of the pod information and the sub-health value in the HealthCheckPolicy health scoring rules. The final deduction for the alert type is calculated based on the alert information and the alert deduction details in the HealthCheckPolicy health scoring rules. A health score is calculated based on the final deductions for the lifecycle type and the alert type.

3. The Kubernetes-based container group health scoring method according to claim 2, characterized in that, Based on the expected container lifetime retention information of the pod and the sub-health value in the HealthCheckPolicy health scoring rules, the final deduction for the lifecycle type is calculated, specifically including the following steps: Create a timer; When the timer reaches the preset time, retrieve all managed Pods; The process involves iterating through the Pods and deducting points for each Pod based on its sub-health value and expected container lifespan retention, resulting in a final deduction for the lifecycle type.

4. The Kubernetes-based container group health scoring method according to claim 3, characterized in that, Each Pod is penalized based on its sub-health score and expected container lifespan retention, resulting in a final lifecycle-type penalty score. This process includes the following steps: To determine if each Pod meets the health score criteria: compare the container startup time with the initial delay time before container startup. If the container startup time is lower than the initial delay time before container startup, then it does not meet the health score criteria. Also, determine the current container status. If the container is in a Terminating or non-Running state, then it does not meet the health score criteria. When a Pod has a health score, the time interval for deducting points is calculated as expected container lifespan / [health value - sub-health value] (periodTime). When the Pod information has a transition time (lastTransitionTime), the deduction value is calculated as [current time - lastTransitionTime] / time interval (periodTime). When the Pod information does not have a transition time (lastTransitionTime), the deduction value is calculated as [current time - container startup time] / time interval (periodTime). Based on this deduction value, the final deduction for the lifecycle type is calculated.

5. The Kubernetes-based container group health scoring method according to claim 4, characterized in that, The calculation process for the final deduction of points for the lifecycle type is as follows: The time interval for deducting points is periodTime: In the formula: S0 represents the time interval for point deduction (periodTime); S0 represents the initial health value; H sub Represents the threshold of sub-health; The points deducted this time are: In the formula: This represents the points deducted in this instance; t represents the current time; Represents the transition time, lastTransitionTime; The final deduction for the lifecycle type is: ； In the formula: The final deduction is for the lifecycle type; w l Represents lifecycle weight.

6. The Kubernetes-based container group health scoring method according to claim 5, characterized in that, Based on the alarm information and the alarm deduction details in the HealthCheckPolicy health scoring rules, the final deduction for the alarm type is calculated, specifically including the following steps: Based on the alarm information, the corresponding pod and its namespace can be parsed out; Retrieve the alarm deduction details from the HealthCheckPolicy health scoring rules corresponding to this pod; Based on the alarm deduction details and alarm information, the corresponding deductions are applied to the pods, and the final deduction for the alarm type is calculated.

7. The Kubernetes-based container group health scoring method according to claim 6, characterized in that, Based on the alarm deduction details and alarm information, corresponding deductions are applied to the pods, and the final deduction for the alert type is calculated using the following formula: Let alarm a be an alarm set A containing alarm information. j The number of times it appears is denoted as c. j Map the single deduction p according to the alarm level j The deduction for a normal alarm is: To avoid significant point deductions caused by alarm storms, an EMA smoothing strategy is introduced: Defining the time constant T j , the smoothing factor: Then the smoothing state a of the alarm intensity j (t) is: in The final deduction for alert type is: If an alarm has healed itself, initiate an exponential decay recovery strategy to allow for slow recovery: Different λ values ​​are configured for different alarm levels. λ can be obtained using the following formula, where r is the decay rate per minute: In the formula: Represents the time interval between the current time and the initial time; : This refers to the total points deducted for alert types. This refers to the point deduction for a specific alarm in alarm set A. This refers to a warning that has self-healed. The deductions calculated using the exponential decay recovery algorithm after a time interval; This refers to the points deducted for each specified scoring strategy within the policy types; Sum the deductions for each strategy. λ refers to the decay rate over the corresponding continuous time period; w a Refers to the alert weight.

8. The Kubernetes-based container group health scoring method according to claim 7, characterized in that, The formula for calculating the health score is as follows: In the formula: Refers to health score; This refers to summing the deductions for each strategy. The final deduction for lifycycle and the final deduction for alert type are added together.

9. The Kubernetes-based container group health scoring method according to claim 8, characterized in that, Restarting the container based on the health score includes the following steps: The health status of a Pod is determined based on the latest health score. A Pod is considered healthy when its health score is above the sub-health threshold; a Pod is considered sub-healthy when its health score is below the sub-health threshold but above the unhealthy threshold; and a Pod is considered unhealthy when its health score is below the unhealthy threshold. When the Pod is in a healthy state, it does not need to be restarted; When a Pod is in a sub-healthy state, check if the time is within the period configured; whether the number of ongoing restart requests in the namespace to which the current Pod belongs has reached the limit; and whether the number of ongoing restart requests in the load to which the current Pod belongs has reached the limit. Restart the container when the Pod is in an unhealthy state.

10. The Kubernetes-based container group health scoring method according to claim 9, characterized in that, To restart a container, the specific steps include: Listen for containerRecreateRequests belonging to this node; Determine if the currently running restart tasks have reached their limit; Determine if there are any containers in the same pod that are already restarting; Gain graceful restart tolerance time; The execPreStopHook executes the operations performed before the application stops. The remaining graceful start / stop time is determined based on the time elapsed after execPreStopHook; if the remaining graceful start / stop time is less than 30 seconds, it is reset to 30 seconds. The StopContainer command is used to delete a container. Write the restart results back to crrstatus. If there is a forced stop during the restart process, set isGrace to false.

Images