Fault handling method and system for physical nodes of a Kubernetes cluster based on BMC

Abstract

The present application provides a method, a system, a computer-readable storage medium, and an electronic device for fault handling of physical nodes in a Kubernetes cluster based on BMC. The method includes: obtaining first status information of a physical node from the BMC set on the physical node; wherein the first status information characterizes the hardware operating status and the operating system operating status of the physical node; then, based on the first status information and a preset custom control policy, processing the physical node with risks or faults. Thereby, the control node actively monitors the hardware operating status and the operating system operating status of the physical node in real time, effectively solving the problems of incomplete and untimely monitoring of risks or faults in the prior art, and after detecting that the physical node has faults or risks, timely processing the physical node according to the preset custom control policy to improve the reliability of the physical node and achieve high availability of the applications on the physical node.

Description

Technical Field

[0001] This application relates to the field of cloud native technologies, and particularly to a method, system, computer-readable storage medium, and electronic device for handling faults of physical nodes in a Kubernetes cluster based on BMC. Background Art

[0002] In the existing related technologies, the method for handling faults of Kubernetes cluster nodes is as follows: through the kubelet component deployed in a containerized manner on each node, the status of the node where it is located is monitored, and the status information of the node where it is located is reported. This is essentially a software-based monitoring method. When this method is used for the status monitoring of physical nodes, when the operating system or other underlying technologies on which the kubelet component depends for normal operation fail, the kubelet component will not be able to work properly, and the Master (control) node will not be able to obtain any status information of the node where the kubelet component is located, and the entire monitoring system will be in a malfunction state. In addition, when there are certain underlying fault risks in a physical node due to hardware problems, these underlying fault risks cannot be perceived by the kubelet component and will not be reported to the Master node. Accordingly, the Master node will not be able to take corresponding measures to handle the fault risks.

[0003] Therefore, how to comprehensively monitor the risks or faults existing in the physical nodes of a Kubernetes cluster and timely handle the physical nodes accordingly to ensure the reliable operation of the applications on the physical nodes in the Kubernetes cluster has become an urgent problem to be solved in the management of the Kubernetes cluster. Summary of the Invention

[0004] The purpose of this application is to provide a method, system, computer-readable storage medium, and electronic device for handling faults of physical nodes in a Kubernetes cluster based on BMC to solve or alleviate the problems existing in the above-mentioned prior art.

[0005] To achieve the above purpose, this application provides the following technical solutions:

[0006] This application provides a method for handling faults of physical nodes in a Kubernetes cluster based on BMC, including: obtaining first status information of the physical node from the BMC set on the physical node; wherein, the first status information characterizes the hardware operation status and the operating system operation status of the physical node;

[0007] According to the first status information, based on a preset custom control policy, handle the physical node with risks or faults.

[0008] In any optional embodiment of the present application, obtaining the first status information of the physical node from the BMC set on the physical node specifically includes:

[0009] Register the BMC set on the physical node, establish a communication connection based on the monitoring protocol, and access the registered BMC on the physical node according to a preset monitoring period to obtain the first status information of the physical node.

[0010] In any optional embodiment of the present application, according to the first status information, based on a preset custom control policy, process the physical node with risks or faults, including:

[0011] According to the first status information, determine the type of risks or faults existing in the physical node;

[0012] According to the type of risks or faults, based on the preset custom control policy, process the physical node.

[0013] In any optional embodiment of the present application, the determining the type of risks or faults existing in the physical node according to the first status information includes:

[0014] When the second status information of the physical node reported by the Kubelet component deployed on the physical node is not received within a preset period, determine that the physical node has a downtime fault according to the hardware operation status and operating system operation status of the physical node characterized by the first status information; wherein, the second status information characterizes the node operation status and container operation status on the physical node.

[0015] In any optional embodiment of the present application, after determining that the physical node has a downtime fault according to the hardware operation status and operating system operation status of the physical node characterized by the first status information, it further includes:

[0016] Obtain the operation log information of the physical node from the BMC set on the physical node;

[0017] Analyze the operation log information of the physical node to determine the reason for the downtime fault of the physical node;

[0018] According to the reason for the downtime fault of the physical node, instruct the BMC set on the physical node to perform a restart operation on the physical node.

[0019] In any optional embodiment of the present application, the determining the type of risks or faults existing in the physical node according to the first status information includes:

[0020] Analyze the first status information to evaluate the impact on the applications deployed on the physical node due to risks or failures of the physical node;

[0021] Classify the types of risks or failures of the physical node according to the impact on the applications deployed on the physical node.

[0022] In any optional embodiment of the present application, the processing of the physical node according to the type of the risk or failure based on the preset custom control policy is specifically as follows:

[0023] Mark a risk label for the physical node with risks; wherein, the risk label includes multiple levels, and the level of the risk label is related to the type of the risk; or,

[0024] Mark a failure label for the physical node with failures; wherein, the failure label includes multiple categories, and the category of the failure label is related to the type of the failure.

[0025] The embodiment of the present application further provides a failure processing system for physical nodes of a Kubernetes cluster based on BMC, including:

[0026] An information acquisition unit configured to acquire the first status information of the physical node from the BMC set on the physical node; wherein, the first status information characterizes the hardware operation status and the operating system operation status of the physical node;

[0027] A policy execution unit configured to process the physical node with risks or failures based on the first status information and a preset custom control policy.

[0028] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and the program is the failure processing method for physical nodes of a Kubernetes cluster based on BMC as described in any one of the above.

[0029] The embodiment of the present application further provides an electronic device, including: a memory, a processor, and a program stored in the memory and executable on the processor, and when the processor executes the program, it implements the failure processing method for physical nodes of a Kubernetes cluster based on BMC as described in any one of the above.

[0030] Compared with the closest prior art, the technical solution of the embodiment of the present application has the following beneficial effects:

[0031] In this application, the control node obtains the first status information of the physical node from the BMC set on the physical node of the Kubernetes cluster; wherein, the first status information characterizes the hardware operation status and the operating system operation status of the physical node; then, based on the first status information and a preset custom control policy, the control node processes the physical node with risks or faults. Thereby, the control node actively monitors the hardware operation status and the operating system operation status of the physical node in real time, effectively solving the problems of incomplete and untimely monitoring of risks or faults in the prior art. After detecting that a physical node has a fault or risk, corresponding processing is performed on the physical node in a timely manner according to the preset custom control policy, so as to improve the reliability of the physical node and achieve high availability of the applications on the physical node. Brief Description of the Drawings

[0032] The accompanying drawings forming a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. Among them:

[0033] Figure 1 is a schematic flowchart of the fault handling process of the existing Kubernetes cluster nodes;

[0034] Figure 2 is a schematic flowchart of a method for handling faults of physical nodes in a Kubernetes cluster based on BMC according to some embodiments of this application;

[0035] Figure 3 is a schematic working principle diagram of a method for handling faults of physical nodes in a Kubernetes cluster based on BMC according to some embodiments of this application;

[0036] Figure 4 is a schematic structural diagram of a system for handling faults of physical nodes in a Kubernetes cluster based on BMC according to some embodiments of this application;

[0037] Figure 5 is a schematic structural diagram of an electronic device according to some embodiments of this application;

[0038] Figure 6 is the hardware structure of an electronic device according to some embodiments of this application. Detailed Embodiments

[0039] The present application will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments. Each example is provided by way of explanation of the present application rather than a limitation thereof. In fact, those skilled in the art will appreciate that modifications and variations can be made to the present application without departing from the scope or spirit thereof. For example, features shown or described as part of one embodiment can be used in another embodiment to yield yet another embodiment. Accordingly, it is desirable that the present application cover such modifications and variations that fall within the scope of the appended claims and their equivalents.

[0040] See Figure 1 , Figure 1 is a schematic flowchart of the failure handling process of existing Kubernetes cluster nodes. In the prior art, the Kubernetes cluster is based on a software-based monitoring method, and the native kubelet component and Scheduler component of the Kubernetes system are used to handle node failures in the cluster. Specifically, the kubelet component on a node in the Kubernetes cluster reports the status information of the node where it is located to the API-Server component on the Master (control) node via the network at regular intervals. If the status information reported by the kubelet component indicates that the status of the node is faulty, or if the Master node does not receive the status information reported by the kubelet component on a certain node for more than a preset duration, the Scheduler component will, according to the Pod (container group) information of that node stored in ETCD, schedule all the Pods on that node to other healthy nodes.

[0041] Using the aforementioned software-based monitoring method for the failure handling of Kubernetes cluster physical nodes has at least the following drawbacks:

[0042] First, it is prone to incorrect scheduling due to misjudgment: when the Master node does not receive the status information reported by the kubelet component on a certain physical node for more than a preset duration, it may be either that there is a jitter or failure in the network between the physical node where the kubelet component is located and the Master node, or that the physical node where the kubelet component is located has failed. Therefore, there is a certain possibility that this solution misjudges a network failure as a physical node failure and wrongly transfers the Pods on a normal physical node.

[0043] Second, the fault handling is not timely: when a physical node fails, it is necessary to immediately redeploy the Pods on this physical node to other healthy physical nodes to ensure the reliability of the Pods in providing services externally. In order to reduce the possibility of misjudging a network fault as a physical node fault in the prior art, a delay determination mechanism is introduced, that is, if the API-Server component on the Master (control) node does not receive the status information of the physical node reported by the kubelet component within a predetermined time, it will not immediately determine this physical node as a faulty node, but continue to wait for the next report from the kubelet component until it still does not receive the status information of the physical node reported by the kubelet component after exceeding the preset maximum duration, then it will determine this physical node as a faulty node. Therefore, in the case where a physical node fails and the kubelet component has reported the information that the physical node where it is located has a fault, if the information reported by the kubelet component about the existence of a fault in the physical node where it is located is lost due to network jitter and cannot reach the Master node, the Master node will continuously wait for the kubelet component to report the status information of the physical node where it is located within a period not exceeding the preset maximum duration. After the information reported by the kubelet component about the existence of a fault in the physical node where it is located successfully reaches the Master node, it will determine this physical node as a faulty node and schedule the Pods on the faulty node, resulting in a delay in the scheduling of the Pods on the faulty node, and the Pods cannot provide services externally normally for a long time.

[0044] Third, it cannot handle serious faults: when there are serious problems such as an operating system fault or downtime in a physical node, the kubelet component will not be able to run normally, and thus will not be able to report the status information of the physical node where it is located, making the Master node unable to know the status information of this physical node, unable to determine the cause of the physical node fault, and even less able to solve the fault of this physical node and restore it to normal.

[0045] Fourth, it cannot handle the risks caused by hardware in physical nodes: when there are underlying fault risks in a physical node due to hardware problems, some fault risks will have a certain impact on specific upper-layer applications, and some fault risks will not affect the normal operation of upper-layer applications, but will reduce the overall reliability of the physical node, and these fault risks cannot be perceived by the kubelet component and will not be reported to the Master node, and the Master node will not take corresponding measures to handle the fault risks.

[0046] To solve the above problems, this application provides a method for handling faults of physical nodes in a Kubernetes cluster based on BMC. Figure 2Schematic flow chart of a method for handling faults of physical nodes in a Kubernetes cluster based on BMC according to some embodiments of the present application; Figure 3 Schematic working principle diagram of a method for handling faults of physical nodes in a Kubernetes cluster based on BMC according to some embodiments of the present application; As Figure 2 , Figure 3 shown, the method for handling faults of physical nodes in the Kubernetes cluster based on BMC includes:

[0047] Step S101, obtain first status information of the physical node from the BMC set on the physical node.

[0048] Among them, the first status information characterizes the hardware operating status and the operating system operating status of the physical node.

[0049] BMC (Baseboard Management Controller, baseboard management controller) is a dedicated service processor, which monitors the status of computers, servers, or other hardware drive devices through sensors and other technical means. In the prior art, when the BMC monitors that a computer, server, or other hardware drive device fails, such as power supply timing, fan control, RAID configuration, etc., it sends an alarm message to the administrator to remind the administrator to handle it, and performs corresponding processing on the monitored object according to the received administrator instruction. That is to say, although the existing BMC can monitor the operating status of computers, servers, or other hardware drive devices and send alarm messages when abnormal situations occur, it cannot automatically judge and automatically handle faults, and still requires the administrator to manually handle the faults, and delays the fault handling time.

[0050] In the embodiments of the present application, Figure 3 as shown, the Kubernetes cluster includes multiple physical nodes, which can be specifically divided into control nodes and working nodes according to different functions. The BMC is set on each physical node of the Kubernetes cluster and is independent of the physical node. It can be understood that the BMC, as a system independent of the physical node, can run without relying on other hardware such as the CPU and memory on the physical node. Therefore, by arranging a BMC independent of the node on the physical node, when the physical node has an operating system failure or the application cannot run normally due to a hardware failure, the BMC can still normally obtain the first status information of the physical node where it is located because it runs independently of the hardware and operating system on the physical node, thus solving the serious problems such as the inability to timely know the operating system failure or hardware failure when monitoring based on the kubelet component, and further solving the problem of the inability to timely solve the fault of the physical node.

[0051] In the embodiments of the present application, the first status information characterizes the hardware operating status and the operating system operating status of the physical node, and the above status information is all status information that cannot be obtained or cannot be obtained in a timely manner by the software monitoring method based on the kubelet component. Specifically, sensors are arranged on the physical node, and the BMC monitors the hardware operating status of the physical node where it is located through the preset sensors on the physical node. It can be understood that the hardware information that can be collected by the preset sensors includes but is not limited to: hardware temperature, fan speed, voltage, power status, chassis intrusion. Correspondingly, the first status information includes the external hardware status information such as the fan, power supply, and chassis monitored by the sensors obtained by the BMC. In addition, the first status information may further include the operating system status information obtained by the BMC, that is, the BMC can directly access the BIOS of the physical node to obtain the operating status information of other hardware such as the CPU, memory, and storage and the operating system status information, so as to reflect the operating status of the underlying operating system on which the applications on the physical node depend for normal operation.

[0052] In the embodiments of the present application, the BMC is used to obtain the hardware operating status through the sensors on the physical node, and the BMC directly accesses the BIOS on the physical node to obtain the operating system operating status of the physical node and the operating status of other hardware, so as to effectively supplement the software monitoring method based on the kubelet component in the traditional node fault handling, making the acquisition of the operating status information of the physical node more complete, comprehensive, and timely, and providing a basis for further taking corresponding measures subsequently.

[0053] In some alternative embodiments, the first status information of the physical node is obtained from the BMC set on the physical node. Specifically, the BMC set on the physical node is registered, and a communication connection based on the monitoring protocol is established to access the registered BMC on the physical node according to the preset monitoring period, so as to obtain the first status information of the physical node.

[0054] In the prior art, the related components of the Kubernetes cluster cannot automatically access the BMC deployed on the physical node, so they cannot obtain various operating status information monitored by the BMC in a timely manner and cannot make full use of the functional characteristics of the BMC. To solve the above problems, in the embodiments of the present application, the Operator component on the control node in the Kubernetes cluster is extended as the core controller, and the core controller registers the BMC set on the physical node and establishes a communication connection based on the monitoring protocol to access the registered BMC on the physical node according to the preset monitoring period, so as to obtain the first status information of the physical node.

[0055] In some alternative embodiments, the core controller registers the BMC set on the physical node. Specifically, the core controller registers the access IP and access password of the BMC set on the physical node, and establishes a communication connection based on a monitoring protocol to allow the core controller to access the BMC set on the physical node according to a preset monitoring period and obtain the first status information of the physical node. It can be understood that the monitoring protocol can be a protocol capable of transmitting the status information of the physical node, such as the redfish API protocol, the IPMI protocol, etc.

[0056] Specifically, the core controller includes: an Operator component. The Operator component is a controller in the Kubernetes cluster that senses the application status. It automatically creates, manages, and configures application instances by extending the Kubernetes API. The Operator component can extend new application resources based on third-party resources and ensure that the application is in the expected state through the controller. In the embodiments of the present application, the Operator component is used to register the BMC set on the physical node and actively access the registered BMC on each physical node to obtain the first status information of the physical node. Specifically, when registering the BMC set on the physical node, the Operator component registers the access IP and access password of the BMC on each physical node, establishes a communication based on the redfish API or the IPMI protocol, and accesses the registered BMC on the physical node according to a preset monitoring period to perform periodic information collection to obtain the first status information of the physical node.

[0057] By extending the Operator component on the control node in the Kubernetes cluster as the core controller to register the BMC set on the physical node, the core controller can periodically access the BMC on the physical node to timely obtain the first status information of the physical node. Thereby, making full use of the functional characteristics of the BMC that can independently monitor the hardware operating state and the operating system operating state continuously and automatically for the physical node, and then enabling the Kubernetes cluster to timely, comprehensively, and accurately master the operating state of the physical node, and formulating and executing corresponding processing strategies based on this.

[0058] Step S102: According to the first status information, based on a preset custom control strategy, process the physical node with risks or faults.

[0059] In the embodiments of the present application, the preset custom control strategy refers to a control strategy predefined by the user for different risk or fault states of the physical node. For example, for a physical node downtime, the physical node can be restarted, or the application on the physical node can be scheduled to other physical nodes.

[0060] In the embodiments of the present application, the preset custom control policy is implemented through the core controller on the control node. Specifically, the core controller includes: a Policy Custom Resource (Policy CRD), which provides a way for users to customize policies to pre-define the corresponding control policies for instructing the core controller to execute when a physical node has risks or failures.

[0061] In the embodiments of the present application, the core controller further includes an extended Scheduler plugin for executing the preset custom control policy. Specifically, the Scheduler plugin is obtained by extending the original scheduling logic of Kubernetes, and it can automatically process physical nodes with risks or failures based on the indication of the preset custom control policy according to the first status information.

[0062] In some alternative embodiments, the control node processes physical nodes with risks or failures based on the first status information and the preset custom control policy, including: the control node determines the type of risks or failures of the physical node according to the first status information; and processes the physical node based on the type of risks or failures and the preset custom control policy. Thereby, the Policy Custom Resource of the core controller provides an automatic processing policy for risks or failures on the physical node, and then automatically processes the risks or failures according to the automatic processing policy, avoiding manual operations, improving the processing efficiency, and reducing the risk of processing errors.

[0063] Further, determining the type of risks or failures of the physical node according to the first status information includes: analyzing the first status information to evaluate the impact on the applications deployed on the physical node due to the risks or failures of the physical node; and dividing the type of risks or failures of the physical node according to the impact on the applications deployed on the physical node. Correspondingly, processing the physical node based on the type of risks or failures and the preset custom control policy is specifically: marking a risk label for the physical node with risks; where the risk label includes multiple levels, and the level of the risk label is related to the type of risks; or marking a failure label for the physical node with failures; where the failure label includes multiple categories, and the category of the failure label is related to the type of failures. Process the physical node with risks or failures based on the label of the physical node and the preset custom control policy.

[0064] In an embodiment of the present application, before obtaining the first status information and analyzing the first status information to evaluate the impact on the applications deployed on the physical node due to risks or failures of the physical node, it further includes: labeling the hardware resources required for each application running on the physical node to establish a mapping relationship between the application and the required hardware, and storing it in custom resources or ETCD.

[0065] In another specific scenario, when the control node determines that some hardware of the physical node is abnormal by analyzing the first status information, that is, the physical node has risks due to hardware problems, and this risk cannot be perceived by existing software-based monitoring methods, then based on the hardware corresponding to the risk and the pre-established mapping relationship between the application and the required hardware, evaluate the impact on the applications deployed on the physical node due to risks or failures of the physical node, and classify the types of risks or failures of the physical node according to the impact on the applications deployed on the physical node.

[0066] Furthermore, the control node analyzes the first status information and determines that partial hardware anomalies have occurred in the physical node, leading to a risk of a decline in the overall performance of the physical node. For example, one of the dual-redundant power supplies on the physical node fails. At this time, the remaining power supply is still operating normally and can supply power to the physical node normally. However, if the remaining power supply also fails, the entire physical node will lose power completely. Therefore, the failure of one of the dual-redundant power supplies increases the probability of the physical node failing; for another example, some of the multiple fans on the physical node are faulty. Although the remaining fans are operating normally and do not affect the normal operation of the physical node, due to the failure of some fans, the remaining fans are in a high-load operation state, resulting in the physical node possibly overheating, and thus the probability of failure increases; for yet another example, there are local failures in the memory, PCI devices, low-speed IO chips and devices, etc. on the physical node, and "Warning" (warning level) log information will be recorded in the system log of the BMC, but the physical node can still operate normally at the operating system level and has no impact on the operation of the application. At this time, based on the analysis of the first status information, the control node determines that the above hardware risks have no impact on the applications on the physical node and determines that the risk type on the physical node is: minor risk, and the level of the risk label corresponding to this type of risk is sub-healthy. When a minor risk occurs, the kubelet component cannot detect the above risk. The control node makes full use of the functional characteristics of the BMC independent of the physical node, accesses the BMC to obtain the first status information, learns that the physical node has a minor risk, and then reduces the scheduling priority and score of the physical node, so that the application is preferentially deployed to other physical nodes. Specifically, the Operator component analyzes the obtained first status information to determine that the physical node has a risk of a decline in overall performance due to partial hardware anomalies. According to the pre-established mapping relationship between the application and the required hardware, when it is determined that the risk does not affect the normal operation of the applications on the physical node, the Operator component labels the physical node with a sub-healthy label to reflect the state of the physical node with a decline in overall performance and an increased probability of failure. The Scheduler plugin, based on the preset custom control policy, performs corresponding deduction calculations on the scores of physical nodes with risk labels of different levels when selecting a physical node for application deployment, and deploys the application on the physical node with the highest score in the cluster, so that the application is preferentially deployed to other physical nodes.

[0067] In the embodiment of the present application, when partial hardware of the physical node has anomalies, resulting in risks for the physical node, and such anomalies cannot be detected based on the software monitoring method, by timely obtaining the first status information from the BMC, analyzing it, and then based on the preset custom control policy, timely and effective responses are made, reducing the impact brought by the risks of the physical node.

[0068] In another scenario, the control node analyzes the first status information. When it determines that a local hardware failure has occurred in a physical node, it labels the faulty physical node with different types of fault labels according to the type of the local hardware failure. For example, if there is a local failure in the disk managed by the RAID card on the physical node or the external memory card of the PCIe, according to the pre-established mapping relationship between the application and the required hardware, it is determined that this type of local failure will only affect the stateful applications stored locally and the applications using the external memory card, and has no impact on other types of applications. Then, the physical node can be labeled with a storage fault label. Another example is that if the GPU on the physical node fails, according to the pre-established mapping relationship between the application and the required hardware, it is determined that this failure will only affect the applications related to image processing and artificial intelligence, and has no impact on other applications that do not use the GPU. Then, the physical node can be labeled with a GPU fault label. At this time, the control node analyzes the first status information and determines that the above local hardware failure affects some applications on the physical node, and labels the physical node with the corresponding type of fault label. The kubelet component cannot detect the above risks. Based on the functional characteristics of the BMC independent of the physical node, the control node can still access the BMC to obtain the first status information, learn that there is a local hardware failure in the physical node, and then schedule the affected applications on the physical node to other physical nodes, and reject the deployment of the corresponding type of applications on this physical node. Specifically, the Operator component analyzes the obtained first status information to determine that some hardware on the physical node has failed. According to the pre-established mapping relationship between the application and the required hardware, when it determines that the local hardware failure affects the corresponding type of applications, the Operator component labels the physical node with the corresponding type of fault label (i.e., the custom taint label) to reflect that the abnormal hardware on the physical node cannot provide services to the corresponding type of applications. The Scheduler plugin, according to the fault labels labeled on the physical node and based on the preset custom control policy, when selecting a physical node for deploying the corresponding type of applications, first filters out the physical nodes with the corresponding type of fault labels from the alternative nodes in advance, and then calculates the scores of the remaining physical nodes, and deploys the application on the physical node with the highest score in the cluster, thereby rejecting the deployment of the corresponding type of applications on this physical node.

[0069] In the embodiments of the present application, for the types of local hardware failures that occur on the physical node, according to the mapping relationship between the application and the hardware resources, different types of local hardware failures are processed separately, which improves the pertinence of the management of local hardware failures in cluster management and makes the scheduling strategy on the physical node more refined and reasonable.

[0070] In another alternative embodiment, determining the type of risk or fault of a physical node according to the first status information includes: when the second status information of the physical node reported by the Kubelet component deployed on the physical node is not received within a preset period, determining that the physical node has a downtime fault according to the hardware operation status and the operating system operation status of the physical node characterized by the first status information. Wherein, the second status information characterizes the node operation status and the container operation status on the physical node.

[0071] Specifically, when the physical node has a downtime due to a CPU fault, a memory fault, or an operating system fault, the Kubelet component cannot operate normally due to the fault of the physical node and cannot report the second status information of the physical node to the Master node within a preset period. The core controller on the Master node actively accesses the BMC on the physical node through the Operator component. The BMC determines that the physical node is in a downtime state by directly detecting the status of the BIOS watchdog on the physical node, and the Operator component labels the downtime fault label for the physical node.

[0072] Further, after determining that the physical node has a downtime fault according to the hardware operation status and the operating system operation status of the physical node characterized by the first status information, it further includes: obtaining the operation log information of the physical node from the BMC set on the physical node; analyzing the operation log information of the physical node to determine the reason for the downtime fault of the physical node; according to the reason for the downtime fault of the physical node, based on a preset custom control policy, instructing the BMC set on the physical node to perform a restart operation on the physical node.

[0073] In the embodiment of the present application, after determining that the physical node is in a downtime state, the core controller on the control node first obtains the operation log information of the physical node from the BMC set on the physical node through the Operator component, analyzes the operation log information of the physical node recorded in the system log of the BMC to determine the reason for the downtime fault of the physical node, and then performs corresponding processing on the physical node according to the reason for the downtime fault of the physical node: when it is determined that the downtime fault of the physical node only requires a system reset of the physical node to be restored, according to the "automatic recovery" function configured in the preset control policy in the core controller, instruct the BMC set on the physical node to perform a restart operation on the physical node. When it is determined that the physical node cannot be restored to normal by performing a restart operation, according to the scheduling policy configured in the preset custom control policy, schedule the applications running on the physical node to a healthy physical node. If there is no idle healthy node available for scheduling when performing application scheduling, include the standby node in the Kubernetes cluster management and schedule the applications to the standby node.

[0074] In the embodiments of the present application, the Scheduler plugin automatically processes physical nodes with risks or failures according to the state of the physical nodes determined by the Operator component and the preset custom control policy. For example, when the Operator component determines that a physical node has a downtime failure and analyzes the operation log information of the physical node on the BMC to obtain the cause of the downtime failure of the physical node, the Scheduler plugin performs automatic processing according to the cause of the downtime failure and the indication of the "automatic recovery" function or scheduling policy in the preset custom control policy. Specifically, when it is determined that the physical node can return to normal through restart, the Scheduler plugin restarts the physical node through the BMC. When it is determined that the physical node cannot return to normal through restart, the Scheduler plugin schedules the applications on the physical node to healthy physical nodes according to the scheduling policy.

[0075] In the embodiments of the present application, when serious problems such as downtime or operating system failure occur to a physical node, the kubelet component will be unable to report the above failure status information. At this time, the control node can obtain the first status information through the BMC, timely learn about the operating state of the physical node, and determine the cause of the physical node failure through the analysis of the first status information, and then perform automatic processing in a timely manner, so as to solve the failure of the physical node and make it return to normal quickly.

[0076] It should be noted that when the control node does not receive the second status information of the physical node reported by the Kubelet component deployed on the physical node within the preset period, it is not necessarily because the physical node has a downtime failure, and corresponding processing is also required according to the first status information of the physical node. For example, the following two specific scenarios:

[0077] In a specific scenario, when the control node does not receive the second status information of the physical node reported by the kubelet component deployed on the physical node within a preset period, the control node is also unable to obtain the first status information from the BMC on the physical node, then it is determined that the physical node has a power failure, and all applications on the physical node are severely affected. At this time, the control node determines that the type of failure of the physical node is a power failure and labels the physical node with a power failure label. Specifically, when the control node does not receive the second status information of the physical node reported by the Kubelet component deployed on the physical node within a preset period, the Operator component on the control node actively accesses the BMC on the physical node and finds that it is also unable to obtain the first status information from the BMC on the physical node, indicating that the physical node has a power failure. At this time, all applications on the physical node are affected, and the Operator component labels the physical node with a power failure label. The Scheduler plugin schedules the applications running on the physical node to healthy nodes based on the preset custom control policy according to the power failure label corresponding to the physical node. If there are no idle healthy nodes available for scheduling, the standby node is incorporated into the Kubernetes cluster management, and the applications on the physical node are scheduled to the standby node.

[0078] In another specific scenario, when the control node does not receive the second status information of the physical node reported by the kubelet component deployed on the physical node within a preset period, the control node obtains the first status information from the BMC on the physical node, analyzes the first status information, determines that the physical node has no risks and failures, the type of failure is a network failure, and all applications on the physical node are not affected, then the control node labels the physical node with a network failure label. Specifically, when the Operator component in the control node does not receive the second status information of the physical node reported by the kubelet component deployed on the physical node within a preset period, the Operator component on the control node actively accesses the BMC on the physical node, obtains the first status information from the BMC on the physical node, analyzes the first status information, determines that the operating system of the physical node is running normally, and the kubelet component has temporarily lost contact with the control node due to a network failure, then the Operator component labels the physical node with a network failure label. The Scheduler plugin does not schedule the applications on the physical node based on the preset custom control policy according to the network failure label corresponding to the physical node, and maintains the original state, waiting for the network to recover. Thereby, when the control node cannot obtain the second status information of the physical node in time, the operating status of the physical node is confirmed through the first status information obtained from the BMC, so as to avoid misjudging the operating status of the physical node, and further prevent scheduling the applications on healthy physical nodes, saving system resources.

[0079] Understandably, in addition to the risks or failure types listed above, the technical solution of the present application can also handle other types of risks or failures on physical nodes. Specifically, when it is necessary to handle new failure or risk types, for the new failure or risk types, the control strategy is defined through the core controller, and then the core controller processes the new failure or risk types accordingly according to the customized control strategy.

[0080] In the embodiment of the present application, the hardware operation status and the operating system operation status of the physical node are obtained through the BMC, and the node operation status and the container operation status on the physical node are obtained through the kubelet component, providing comprehensive and timely operation status information for the Kubernetes cluster. The Kubernetes cluster automatically processes the failures or risks of the physical node by expanding the control node and according to the obtained physical node operation status information, thus solving the problems of incomplete and untimely monitoring of the operation status of the physical node and inability to automatically process failures in the prior art, saving labor costs and improving the processing efficiency.

[0081] In the embodiment of the present application, the Operator component on the control node in the Kubernetes cluster is expanded as the core controller to register the BMC set on the physical node, so that the core controller can periodically access the BMC on the physical node to obtain the first status information of the physical node in a timely manner. Thereby, making full use of the functional characteristics that the BMC can independently monitor the hardware operation status and the operating system operation status continuously and automatically for the physical node, and then enabling the Kubernetes cluster to timely, comprehensively and accurately master the operation status of the physical node, and formulating and executing corresponding processing strategies based on this.

[0082] In the embodiment of the present application, when it is determined that the physical node has a downtime failure based on the physical node operation status information obtained through the BMC, according to the preset customized control strategy, the BMC is instructed to automatically restart the physical node with the downtime failure, so that the physical node can be restored in a timely manner.

[0083] Exemplary system

[0084] Figure 4 FIG. is a schematic structural diagram of a failure processing system for a physical node of a Kubernetes cluster based on BMC according to some embodiments of the present application; as Figure 4As shown in the figure, the system includes: an information acquisition unit 401 and a policy execution unit 402. The information acquisition unit 401 is configured to obtain the first status information of the physical node from the BMC set on the physical node; wherein, the first status information characterizes the hardware operation status and the operating system operation status of the physical node; the policy execution unit 402 is configured to process the physical node with risks or faults based on the preset custom control policy according to the first status information.

[0085] In some alternative embodiments, the information acquisition unit 401 is further configured to: register the BMC set on the physical node, establish a communication connection based on the monitoring protocol, and access the registered BMC on the physical node according to the preset monitoring period to obtain the first status information of the physical node.

[0086] In some alternative embodiments, the policy execution unit 402 is further configured to: determine the type of risks or faults of the physical node according to the first status information; and process the physical node based on the preset custom control policy according to the type of risks or faults.

[0087] In some alternative embodiments, determining the type of risks or faults of the physical node according to the first status information includes: when the second status information of the physical node reported by the Kubelet component deployed on the physical node is not received within the preset period, determining that the physical node has a downtime fault according to the hardware operation status and the operating system operation status of the physical node characterized by the first status information; wherein, the second status information characterizes the node operation status and the container operation status on the physical node.

[0088] In some alternative embodiments, after determining that the physical node has a downtime fault according to the hardware operation status and the operating system operation status of the physical node characterized by the first status information, it further includes: obtaining the operation log information of the physical node from the BMC set on the physical node; analyzing the operation log information of the physical node to determine the reason for the downtime fault of the physical node; and instructing the BMC set on the physical node to perform a restart operation on the physical node according to the reason for the downtime fault of the physical node.

[0089] In some alternative embodiments, determining the type of risks or faults of the physical node according to the first status information includes: analyzing the first status information to evaluate the impact on the applications deployed on the physical node due to the risks or faults of the physical node; and classifying the type of risks or faults of the physical node according to the impact on the applications deployed on the physical node.

[0090] In some alternative embodiments, according to the first status information and based on a preset custom control policy, physical nodes with risks or faults are processed as follows: a risk label is marked on a physical node with risks; wherein, the risk label includes multiple levels, and the level of the risk label is related to the type of risk; or, a fault label is marked on a physical node with faults; wherein, the fault label includes multiple categories, and the category of the fault label is related to the type of fault.

[0091] The fault handling system for physical nodes of a Kubernetes cluster based on BMC provided by the embodiments of the present application can implement the steps and processes of any of the above embodiments of the fault handling method for physical nodes of a Kubernetes cluster based on BMC, and achieve the same beneficial effects, which will not be elaborated herein one by one.

[0092] Exemplary device

[0093] Figure 5 FIG. is a schematic structural diagram of an electronic device provided according to some embodiments of the present application; as Figure 5 shown, the electronic device includes:

[0094] One or more processors 501;

[0095] A computer-readable medium that can be configured to store one or more programs 502. When the one or more processors 501 execute the one or more programs 502, the following steps are implemented:

[0096] Obtain the first status information of the physical node from the BMC set on the physical node; wherein, the first status information characterizes the hardware operating status and the operating system operating status of the physical node; according to the first status information and based on a preset custom control policy, process the physical node with risks or faults.

[0097] Figure 6 FIG. is the hardware structure of an electronic device provided according to some embodiments of the present application, as Figure 6 shown, the hardware structure of the electronic device may include: a processor 601, a communication interface 602, a computer-readable medium 603, and a communication bus 604.

[0098] Wherein, the processor 601, the communication interface 602, and the computer-readable medium 603 complete communication with each other through the communication bus 604.

[0099] Optionally, the communication interface 602 may be an interface of a communication module, such as an interface of a GSM module.

[0100] Wherein, the processor 601 may be specifically configured to:

[0101] Obtain the first status information of the physical node from the BMC set on the physical node; wherein, the first status information characterizes the hardware operating status and the operating system operating status of the physical node; according to the first status information, based on a preset custom control policy, process the physical node with risks or faults.

[0102] The processor can be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc., and can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0103] The electronic devices in the embodiments of the present application exist in various forms, including but not limited to:

[0104] (1) Mobile communication devices: The characteristic of such devices is that they have mobile communication functions and mainly aim to provide voice and data communication. Such terminals include: smart phones (e.g., iPhone), multimedia phones, functional phones, and low-end phones, etc.

[0105] (2) Ultra-mobile personal computer devices: Such devices belong to the category of personal computers, have computing and processing functions, and generally also have the characteristic of mobile Internet access. Such terminals include: PDA, MID, and UMPC devices, etc., such as iPad.

[0106] (3) Portable entertainment devices: Such devices can display and play multimedia content. Such devices include: audio and video players (e.g., iPod), handheld game consoles, e-books, and smart toys and portable in-vehicle navigation devices.

[0107] (4) Servers: Devices that provide computing services. The composition of a server includes a processor, a hard disk, a memory, a system bus, etc. A server is similar to a general computer architecture, but due to the need to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability, etc.

[0108] (5) Other electronic devices with data interaction functions.

[0109] It should be noted that, according to the implementation requirements, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0110] The method according to the embodiments of the present application described above can be implemented in hardware, firmware, or be implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or be implemented as computer code originally stored in a remote recording medium or a non-transitory machine storage medium and downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component (such as a RAM, a ROM, a flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the fault handling method of the physical node of the BMC-based Kubernetes cluster described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown herein.

[0111] Those of ordinary skill in the art can realize that the units and method steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application of the technical solution and the involved constraints. A professional technician can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0112] It should be noted that the embodiments in this specification are all described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and the key points of each embodiment are the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the corresponding parts of the method embodiments for the relevant content. The device and system embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components referred to as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative work.

[0113] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A fault handling method for physical nodes of a Kubernetes cluster based on BMC, characterized in that Deploy core controllers on the control nodes of the Kubernetes cluster. The core controllers include an extended Operator component, policy custom resources, and an extended Scheduler plugin. The policy custom resources provide a way for users to customize control policies. The method includes: The extended Operator component registers the BMC set on the physical node, establishes a communication connection based on the monitoring protocol, and accesses the registered BMC on the physical node at a preset monitoring period to obtain the first status information of the physical node. Among them, the first status information characterizes the hardware operation status and the operating system operation status of the physical node. The extended Scheduler plugin performs scheduling processing on the physical node with risks or faults based on the first status information and the preset custom control policies in the policy custom resources, including: The extended Scheduler plugin determines the type of risks or faults existing in the physical node according to the first status information; The extended Scheduler plugin performs scheduling processing on the physical node based on the preset custom control policies in the policy custom resources according to the type of risks or faults. The extended Scheduler plugin determines the type of risks or faults existing in the physical node according to the first status information, including: When the extended Scheduler plugin does not receive the second status information of the physical node reported by the Kubelet component deployed on the physical node within a preset period, it determines that the physical node has a downtime fault according to the hardware operation status and the operating system operation status of the physical node characterized by the first status information. Among them, the second status information characterizes the node operation status and the container operation status on the physical node.

2. The fault handling method for the physical node of the Kubernetes cluster based on BMC according to claim 1, wherein After determining that the physical node has a downtime fault according to the hardware operation status and the operating system operation status of the physical node characterized by the first status information, it further includes: Obtain the operation log information of the physical node from the BMC set on the physical node; Analyze the operation log information of the physical node to determine the reason for the downtime fault of the physical node; According to the reason for the downtime fault of the physical node, instruct the BMC set on the physical node to perform a restart operation on the physical node.

3. The fault handling method for the physical node of the Kubernetes cluster based on BMC according to claim 1, wherein, The extended Scheduler plugin determines the type of risks or faults existing in the physical node according to the first status information, and further includes: Analyze the first status information to evaluate the impact on the applications deployed on the physical node due to the risks or faults existing in the physical node; Divide the type of risks or faults existing in the physical node according to the impact on the applications deployed on the physical node.

4. The fault handling method for the physical node of the Kubernetes cluster based on BMC according to claim 1 or 3, characterized in that, The extended Scheduler plugin customizes the custom control policy preset in the policy custom resource based on the type of the risk or fault, and performs scheduling processing on the physical node, specifically as follows: Label the physical node with a risk where there is a risk; wherein, the risk label includes multiple levels, and the level of the risk label is related to the type of the risk; or, Label the physical node with a fault where there is a fault; wherein, the fault label includes multiple categories, and the category of the fault label is related to the type of the fault.

5. A fault handling system for physical nodes of a Kubernetes cluster based on BMC, characterized in that, A core controller is deployed on the control node of the Kubernetes cluster. The core controller includes an extended Operator component, a policy custom resource, and an extended Scheduler plugin. The policy custom resource provides a way for users to customize control policies. The system includes: An information acquisition unit configured to register the BMC set on the physical node by the extended Operator component, establish a communication connection based on a monitoring protocol, and access the registered BMC on the physical node according to a preset monitoring period to obtain first status information of the physical node; wherein, the first status information characterizes the hardware operation status and the operating system operation status of the physical node. A policy execution unit configured to, according to the first status information, the extended Scheduler plugin performs scheduling processing on the physical node with a risk or a fault based on the custom control policy preset in the policy custom resource. The policy execution unit is further configured to: The extended Scheduler plugin determines the type of risk or fault existing in the physical node according to the first status information; The extended Scheduler plugin customizes the custom control policy preset in the policy custom resource based on the type of the risk or fault, and performs scheduling processing on the physical node; The extended Scheduler plugin determines the type of risk or fault existing in the physical node according to the first status information, including: when the extended Scheduler plugin does not receive the second status information of the physical node reported by the Kubelet component deployed on the physical node within a preset period, according to the hardware operation status and the operating system operation status of the physical node characterized by the first status information, it is determined that the physical node has a downtime fault; wherein, the second status information characterizes the node operation status and the container operation status on the physical node.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is the fault handling method for the physical node of the Kubernetes cluster based on BMC according to any one of claims 1-4.

7. An electronic device, characterized in that, Including: A memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the fault handling method for the physical node of the Kubernetes cluster based on BMC according to any one of claims 1-4.

Images