Method and device for repairing fault node of container arrangement system and storage medium

Abstract

The invention discloses a method and a device for repairing a fault node of a container arrangement system, and a storage medium. Relates to the field of cloud computing, and the method comprises the steps: obtaining a monitoring index of a container arrangement system for a target node in a cluster, the monitoring index being measurement data used for describing an operation index of the target node; under the condition that the target node is determined to be a fault node according to the monitoring index and a predefined alarm rule, generating first alarm information according to the monitoring index and the alarm rule; generating target alarm information according to the first alarm information through the alarm manager, and sending the target alarm information to an event callback mechanism; and under the condition that the event callback mechanism receives the target alarm information, calling an automatic operation and maintenance tool to repair the fault node. Through application of the method and the device, the problem of low repair efficiency due to dependence on manual repair when the node in the container arrangement system cluster has a fault in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of cloud computing, and more specifically, to a method, device, and storage medium for repairing a faulty node in a container orchestration system. Background Art

[0002] In the current cloud computing and microservices architecture, containerization has become a key technology for achieving efficient application deployment, efficient resource utilization, and elastic system expansion. Container orchestration systems significantly improve the operation and maintenance efficiency of data centers and the operating environment of applications by scheduling, deploying, and managing containers. These systems can automatically handle the lifecycle management of containers and deploy container applications to multiple nodes in the cluster to ensure high availability and optimized use of resources. However, as the scale and complexity of containerized applications continue to grow, cluster node and component failures are becoming increasingly frequent. For example, node hardware failures, network anomalies, operating system problems, or container service anomalies may cause nodes to fail to work properly, affecting the stability of the entire cluster and the availability of applications. For large-scale cloud-native deployments, it is critical to quickly detect and respond to these failures to reduce business interruption time and ensure service consistency and user experience.

[0003] At present, the monitoring and alarm mechanisms of container orchestration systems are already quite mature. When abnormal conditions are monitored, these systems can send alarm notifications to operation and maintenance personnel through alarm services, reminding them to take immediate action. Although monitoring and alarm mechanisms play an important role in fault detection, the process from fault detection to fault repair in existing technologies is still highly dependent on manual intervention. When an alarm is triggered, operation and maintenance personnel need to manually log in to the faulty node, troubleshoot the cause of the problem, and perform a series of complex repair operations. In this process, there are limitations on the speed of manual response, especially during non-working hours or when facing large-scale clusters, the fault response time may be greatly extended. In addition, reliance on manual repairs may also lead to operational inconsistencies, and sometimes even introduce new problems, reducing the overall reliability and stability of the system.

[0004] In view of the problem that when a node in a container orchestration system cluster fails in the related technology, it relies on manual repair and the repair efficiency is low, and no effective solution has been proposed so far. Summary of the invention

[0005] The main purpose of the present application is to provide a method, device and storage medium for repairing a faulty node in a container orchestration system, so as to solve the problem in the related art that when a node in a container orchestration system cluster fails, it relies on manual repair and has low repair efficiency.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for repairing a faulty node of a container orchestration system is provided. The method comprises: obtaining monitoring indicators of the container orchestration system for a target node in a cluster, wherein the monitoring indicators are measurement data used to describe the operating indicators of the target node, wherein the operating indicators are at least one of the following: resource usage, operating status, and operating performance; when the target node is determined to be a faulty node based on the monitoring indicators and predefined alarm rules, generating a first alarm message based on the monitoring indicators and the alarm rules; generating target alarm messages based on the first alarm messages through an alarm manager, and sending the target alarm messages to an event callback mechanism; when the event callback mechanism receives the target alarm messages, calling an automated operation and maintenance tool to repair the faulty node.

[0007] Furthermore, before generating target alarm information based on the first alarm information through the alarm manager and sending the target alarm information to the event callback mechanism, the method also includes: receiving the first alarm information through the alarm manager, performing a deduplication operation on the first alarm information according to a predefined deduplication rule to obtain second alarm information; grouping the alarm information in the second alarm information according to a grouping strategy; and converting the format of the grouped second alarm information to obtain third alarm information.

[0008] Furthermore, before the alarm manager generates target alarm information based on the first alarm information and sends the target alarm information to the event callback mechanism, the method also includes: performing a priority evaluation on multiple alarm information in the third alarm information through the alarm manager, and assigning a priority to each alarm information in the third alarm information according to a preset priority rule; and generating the target alarm information based on the alarm information with the highest priority in the third alarm information.

[0009] Furthermore, when the event callback mechanism receives the target alarm information, before calling the automated operation and maintenance tool to repair the faulty node, the method also includes: detecting the network connectivity of the faulty node through the event callback mechanism to determine whether the faulty node can be accessed through the network connection; if the faulty node can be accessed, calling the automated configuration file for node repair in the automated operation and maintenance tool to repair the faulty node; if the faulty node cannot be accessed, calling the automated configuration file for node replacement in the automated operation and maintenance tool to repair the faulty node.

[0010] Furthermore, if the faulty node is accessible, the automated configuration file for node repair in the automated operation and maintenance tool is called to repair the faulty node, including: calling the interface of the container orchestration system through the automated operation and maintenance tool to mark the faulty node as an unschedulable state; deleting the deployed container from the faulty node; adding a target label to prevent the faulty node from still generating an alarm during the repair process; deleting the directories and files related to the container orchestration system in the faulty node; reinstalling the container orchestration system components in the faulty node; rejoining the faulty node to the container orchestration system cluster and deleting the target label; setting the faulty node to a schedulable state through the automated operation and maintenance tool to complete the repair of the faulty node.

[0011] Furthermore, if the faulty node is inaccessible, the automated configuration file for node replacement in the automated operation and maintenance tool is called to repair the faulty node, including: deleting the faulty node from the container orchestration system; determining a spare node from the spare node pool of the container orchestration system through an event callback mechanism; adding the spare node to the cluster to replace the faulty node, thereby completing the repair of the faulty node.

[0012] Furthermore, when the event callback mechanism receives the target alarm information, after calling the automated operation and maintenance tool to repair the faulty node, the method also includes: verifying whether the components of the container orchestration system are successfully started at the faulty node through the automated operation and maintenance tool; if successfully started, confirming that the repair of the faulty node is completed.

[0013] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a device for repairing a faulty node in a container orchestration system is provided. The device includes: an acquisition unit, which is used to acquire the monitoring indicators of the target node in the cluster of the container orchestration system, wherein the monitoring indicators are measurement data used to describe the operating indicators of the target node, wherein the operating indicators are at least one of the following: resource usage, operating status, and operating performance; a first generation unit, which is used to generate a first alarm message according to the monitoring indicators and the alarm rules when the target node is determined to be a faulty node according to the monitoring indicators and the predefined alarm rules; a sending unit, which is used to generate target alarm information according to the first alarm information through the alarm manager, and send the target alarm information to the event callback mechanism; a first repair unit, which is used to call the automated operation and maintenance tool to repair the faulty node when the event callback mechanism receives the target alarm information.

[0014] Furthermore, the device also includes: a deduplication unit, which is used to receive the first alarm information through the alarm manager before generating the target alarm information based on the first alarm information through the alarm manager and sending the target alarm information to the event callback mechanism, and perform a deduplication operation on the first alarm information according to a predefined deduplication rule to obtain the second alarm information; a grouping unit, which is used to group the alarm information in the second alarm information according to the grouping strategy; and a conversion unit, which is used to convert the format of the grouped second alarm information to obtain the third alarm information.

[0015] Furthermore, the device also includes: an evaluation unit, which is used to perform priority evaluation on multiple alarm information in the third alarm information through the alarm manager before generating target alarm information based on the first alarm information and sending the target alarm information to the event callback mechanism, and assign a priority to each alarm information in the third alarm information according to a preset priority rule; a second generation unit, which is used to generate target alarm information based on the alarm information with the highest priority in the third alarm information.

[0016] Furthermore, the device also includes: a detection unit, which is used to detect the network connectivity of the faulty node through the event callback mechanism before calling the automated operation and maintenance tool to repair the faulty node when the event callback mechanism receives the target alarm information, so as to determine whether the faulty node can be accessed through the network connection; a second repair unit, which is used to call the automated configuration file for node repair in the automated operation and maintenance tool to repair the faulty node if the faulty node can be accessed; and a third repair unit, which is used to call the automated configuration file for node replacement in the automated operation and maintenance tool to repair the faulty node if the faulty node cannot be accessed.

[0017] Furthermore, the second repair unit includes: a calling module, which is used to call the interface of the container orchestration system through an automated operation and maintenance tool to mark the faulty node as an unschedulable state; a first deletion module, which is used to delete the deployed container from the faulty node; a first addition module, which is used to add a target label to prevent the faulty node from still generating an alarm during the repair process; a second deletion module, which is used to delete directories and files related to the container orchestration system in the faulty node; an installation module, which is used to reinstall the container orchestration system components in the faulty node; a third deletion module, which is used to rejoin the faulty node to the container orchestration system cluster and delete the target label; and a repair module, which is used to set the faulty node to a schedulable state through an automated operation and maintenance tool to complete the repair of the faulty node.

[0018] Furthermore, the third repair unit includes: a fourth deletion module, used to delete the faulty node from the container orchestration system; a determination module, used to determine a spare node from the spare node pool of the container orchestration system through an event callback mechanism; and a second addition module, used to add the spare node to the cluster to replace the position of the faulty node to complete the repair of the faulty node.

[0019] Furthermore, the device also includes: a verification unit, which is used to call the automated operation and maintenance tool to repair the faulty node when the event callback mechanism receives the target alarm information, and then verify whether the components of the container orchestration system are successfully started at the faulty node through the automated operation and maintenance tool; a confirmation unit, which is used to confirm that the repair of the faulty node is completed if it is successfully started.

[0020] According to another aspect of the present application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein when the program is running, a device where the computer-readable storage medium is located is controlled to execute any method for repairing a faulty node in a container orchestration system.

[0021] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing a repair method for a failed node of any container orchestration system.

[0022] According to another aspect of the present application, a computer program product is provided, including computer instructions, which, when executed by a processor, implement the steps of any one of the above-mentioned methods for repairing a failed node in a container orchestration system.

[0023] In an embodiment of the present application, monitoring indicators of a target node in a cluster are obtained by a container orchestration system, wherein the monitoring indicators are measurement data used to describe the operating indicators of the target node, wherein the operating indicators are at least one of the following: resource usage, operating status, and operating performance; when the target node is determined to be a faulty node based on the monitoring indicators and predefined alarm rules, a first alarm message is generated based on the monitoring indicators and the alarm rules; a target alarm message is generated based on the first alarm message through an alarm manager, and the target alarm message is sent to an event callback mechanism; when the event callback mechanism receives the target alarm message, an automated operation and maintenance tool is called to repair the faulty node, thereby solving the technical problem in the related art that when a node in a container orchestration system cluster fails, the repair efficiency is low due to reliance on manual repair. In the present application, by obtaining the monitoring indicators of the target node in the cluster of the container orchestration system, if the target node is determined to be a faulty node based on the monitoring indicators and predefined alarm rules, a first alarm message is generated, and the alarm manager generates target alarm information based on the first alarm information and sends it to the event callback mechanism. After receiving the target alarm information, the event callback mechanism calls the automated operation and maintenance tool to repair the faulty node, thereby achieving the technical effect of improving the efficiency of repairing faulty nodes in the container orchestration system cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 A hardware structure block diagram of a computer terminal for implementing a method for repairing a failed node in a container orchestration system is shown;

[0026] Figure 2 It is a flowchart of a method for repairing a faulty node in a container orchestration system provided according to an embodiment of the present application;

[0027] Figure 3 is a flowchart of deduplication and grouping processing of first alarm information provided according to an embodiment of the present application;

[0028] Figure 4 is a flowchart of priority evaluation of first alarm information provided according to an embodiment of the present application;

[0029] Figure 5 is a flowchart for determining whether a faulty node can be accessed through a network connection according to an embodiment of the present application;

[0030] Figure 6 It is a flowchart of repairing a faulty node by calling an automated configuration file for node repair in an automated operation and maintenance tool provided in an embodiment of the present application;

[0031] Figure 7 It is a flowchart of repairing a faulty node by calling an automated configuration file for node replacement in an automated operation and maintenance tool provided in an embodiment of the present application;

[0032] Figure 8 It is a flowchart for verifying whether a component of a container orchestration system is successfully started according to an embodiment of the present application;

[0033] Fig. 9 It is a schematic diagram of a device for repairing a failed node of a container orchestration system provided according to an embodiment of the present application;

[0034] Fig.10 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] Example 1

[0038] According to an embodiment of the present application, a method embodiment of repairing a faulty node in a container orchestration system is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for repairing a failed node in a container orchestration system is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0040] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0041] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for repairing a faulty node of a container orchestration system in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the method for repairing a faulty node of a container orchestration system described above is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0042] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0043] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0044] Under the above operating environment, this application provides Figure 2 The following is a method for repairing a failed node in a container orchestration system. Figure 2 It is a flowchart of a method for repairing a failed node of a container orchestration system according to Embodiment 1 of the present application.

[0045] Step S201, obtaining monitoring indicators of the container orchestration system for the target node in the cluster, wherein the monitoring indicators are measurement data used to describe the operating indicators of the target node, wherein the operating indicators are at least one of the following: resource usage, operating status, and operating performance.

[0046] Optionally, the container orchestration system refers to a platform that can automatically deploy, schedule and manage containerized applications. The monitoring indicators are a series of data that measure the running status of nodes, including but not limited to resource usage (such as CPU, memory, disk and network usage), running status (such as whether the node is healthy, whether the service is running, etc.) and running performance (such as response time, throughput, etc.). The target node refers to one or more nodes selected for monitoring among a series of nodes managed by the container orchestration system.

[0047] For example, in a container orchestration system cluster, multiple nodes are monitored to run different containerized services. The collection tool will collect indicators such as CPU usage, memory utilization, disk space usage, and network latency for each node (corresponding to the above monitoring indicators). The target node's CPU usage is 80%, memory utilization is 75%, disk space is 20%, and the network latency from the target node to other nodes in the cluster is 1 millisecond.

[0048] Step S202: when the target node is determined to be a faulty node according to the monitoring indicator and the predefined alarm rule, first alarm information is generated according to the monitoring indicator and the alarm rule.

[0049] Optionally, the above alarm rules are a set of logical conditions for defining when to trigger an alarm. The alarm rules can be based on the threshold or trend of the monitoring indicator, such as triggering an alarm when the CPU usage exceeds 90% for 5 minutes. In this step, based on the monitoring indicator data collected by the collection tool and the predefined alarm rules, it can be automatically determined whether the target node is in a faulty state. If the monitoring indicator exceeds the threshold set by the alarm rule, a first alarm message will be generated. The first alarm message may include detailed information such as the identification of the faulty node, the alarm type, the trigger time, the severity of the alarm, etc.

[0050] For example, the alarm rule of the container orchestration system cluster defines that when the CPU usage of the target node exceeds 85% and lasts for more than 5 minutes, the target node is considered a faulty node. After determining that the target node is a faulty node, a first alarm message may be generated, and the first alarm message may indicate that the CPU usage of the faulty node is abnormal.

[0051] Step S203: generating target alarm information according to the first alarm information through the alarm manager, and sending the target alarm information to the event callback mechanism.

[0052] Optionally, the alarm manager is responsible for receiving, processing and forwarding alarm information (such as the first alarm information, target alarm information, etc.). The event callback mechanism is an event-driven mechanism that allows the system to send an HTTP request to a predefined URL when a specific event occurs (such as an alarm trigger) to notify or call an automation tool to perform a specific task. After receiving the first alarm information, the alarm manager can further process the alarm information, perform operations such as deduplication, grouping, and silencing, to generate more refined and specific target alarm information.

[0053] Step S204: When the event callback mechanism receives the target alarm information, the automated operation and maintenance tool is called to repair the faulty node.

[0054] Optionally, after receiving the target alarm information, the above-mentioned automated operation and maintenance tool can call a preset repair script to perform the repair operation according to the fault type and repair strategy in the target alarm information. These repair scripts define a standardized repair process to ensure the consistency and reliability of the repair operation.

[0055] In summary, the above technical solutions demonstrate the repair process of faulty nodes in the container orchestration system, from monitoring to fault detection, alarm generation, alarm information processing, and then to automated fault recovery. This repair process not only reduces the need for manual intervention, but also improves the efficiency and accuracy of system fault response.

[0056] The method for repairing a faulty node in a container orchestration system provided in an embodiment of the present application obtains monitoring indicators of a target node in a cluster by the container orchestration system, wherein the monitoring indicators are measurement data used to describe the operating indicators of the target node, wherein the operating indicators are at least one of the following: resource usage, operating status, and operating performance; when the target node is determined to be a faulty node based on the monitoring indicators and predefined alarm rules, a first alarm message is generated based on the monitoring indicators and the alarm rules; a target alarm message is generated based on the first alarm message by an alarm manager, and the target alarm message is sent to an event callback mechanism; when the event callback mechanism receives the target alarm message, an automated operation and maintenance tool is called to repair the faulty node, thereby solving the technical problem in the related art that when a node in a cluster of a container orchestration system fails, manual repair is relied upon and the repair efficiency is low. In the present application, by obtaining the monitoring indicators of the target node in the cluster of the container orchestration system, if the target node is determined to be a faulty node based on the monitoring indicators and predefined alarm rules, a first alarm message is generated, and the alarm manager generates target alarm information based on the first alarm information and sends it to the event callback mechanism. After receiving the target alarm information, the event callback mechanism calls the automated operation and maintenance tool to repair the faulty node, thereby achieving the technical effect of improving the efficiency of repairing faulty nodes in the container orchestration system cluster.

[0057] In an optional embodiment, if Figure 3 As shown, before the alarm manager generates target alarm information according to the first alarm information and sends the target alarm information to the event callback mechanism, the method further includes:

[0058] Step S301: receiving first alarm information through an alarm manager, performing a deduplication operation on the first alarm information according to a predefined deduplication rule, and obtaining second alarm information.

[0059] Optionally, the above predefined deduplication rules are used in the alarm manager to merge or filter duplicate alarm information to avoid unnecessary alarm notifications and waste of resources. The deduplication rules can be alarm identifiers, time windows, alarm states, etc.

[0060] For example, if the CPU usage of the target node in the cluster is detected to exceed the threshold continuously, multiple first alarm messages are generated, which may contain repeated fault descriptions. These similar alarm messages can be merged into one according to predefined deduplication rules, such as time windows, to generate a second alarm message. This process avoids multiple processing of the same problem.

[0061] Step S302: group the alarm information in the second alarm information according to a grouping strategy.

[0062] Optionally, the above grouping strategy is a logical rule used to classify alarm information according to factors such as type, source, priority, etc.

[0063] For example, there are multiple types of alarm information in the second alarm information, including high CPU usage, high memory usage, insufficient disk space, abnormal network connection, etc. According to the grouping strategy, alarm information about resource usage (such as CPU and memory) is grouped together, and alarm information related to storage (such as insufficient disk space) is grouped together. Through grouping, the distribution of fault nodes of different categories can be more clearly understood, which facilitates the subsequent invocation of specific repair scripts for repair as needed.

[0064] Step S303: convert the format of the grouped second alarm information to obtain third alarm information.

[0065] For example, the second alarm information can be converted into a format that can be recognized by the event callback mechanism, and the converted third alarm information can include the identifier of the faulty node and the fault type, and can also include a detailed description of the fault, the triggering time, the priority, and the expected repair operation, etc. In this way, when the event callback mechanism receives the third alarm information, the corresponding repair process can be triggered immediately without additional parsing or format conversion.

[0066] In summary, the above technical solution, by adding deduplication, grouping and format conversion steps in the alarm manager, the deduplication rules avoid repeated processing of the same fault, the grouping strategy makes the alarm information processing more orderly, and the format conversion ensures the seamless transmission of information between different components, providing clear and standard input for automated operation and maintenance tools, thereby realizing an automated chain from monitoring alarms to fault repair and reducing fault recovery time.

[0067] In an optional embodiment, if Figure 4 As shown, before the alarm manager generates target alarm information according to the first alarm information and sends the target alarm information to the event callback mechanism, the method further includes:

[0068] Step S401: perform priority evaluation on multiple alarm information in the third alarm information through an alarm manager, and assign a priority to each alarm information in the third alarm information according to a preset priority rule.

[0069] Optionally, the above-mentioned preset priority rules are a set of preset logical criteria for evaluating and sorting the priorities of alarms or events. Priority rules ensure that the system can identify and prioritize those faults that have the greatest impact on the business, are the most urgent, or are most likely to require an immediate response. Priority rules can include elements: severity, scope of impact, business criticality, and so on. Severity is the most intuitive element and is proportional to the degree of impact on the system or business. For example, alarms such as CPU, memory, or disk usage exceeding a threshold, network connectivity being interrupted, and major services being unavailable may be marked as "Critical" or "High", while some minor performance fluctuations or potential system problems may be marked as "Warning" or "Info".

[0070] For example, the third alarm information includes the following alarm information: Node A: CPU usage exceeds the threshold, the severity level is "Critical", Node B: memory usage exceeds the threshold, the severity level is "High", Node C: network connection is unstable, the severity level is "Warning". These alarm information can be evaluated and sorted according to the preset priority rules. The rules can set the "Critical" alarm to have the highest priority, followed by "High", and "Warning" to have the lowest priority.

[0071] Step S402: Generate target alarm information according to the alarm information with the highest priority in the third alarm information.

[0072] For example, after completing the priority evaluation in step S401, it can be identified that the high CPU usage of Node A is the most serious alarm, and therefore has the highest priority. Target alarm information can be generated based on the alarm information of Node A.

[0073] In summary, through the above steps S401 and S402, priority evaluation can ensure that critical faults are handled first, and reasonable resource allocation decisions can be made when facing different types of faults.

[0074] In an optional embodiment, if Figure 5 As shown, when the event callback mechanism receives the target alarm information, before calling the automated operation and maintenance tool to repair the faulty node, the method further includes:

[0075] Step S501 : detecting the network connectivity of the faulty node through an event callback mechanism to determine whether the faulty node can be accessed through the network connection.

[0076] For example, if the target alarm message is that the CPU usage of Node D is too high, after the event callback mechanism receives this alarm message, the event callback mechanism will try to access Node D through the network connection to confirm whether the node can be repaired through remote operation. This can be done through a simple ping test, SSH connection attempt, or HTTP request to check whether Node D responds.

[0077] Step S502: If the faulty node is accessible, call the automatic configuration file for node repair in the automatic operation and maintenance tool to repair the faulty node.

[0078] For example, if the network connectivity check of Node D succeeds, the event callback mechanism can call the node repair automation configuration file in the automated operation and maintenance tool and pass the detailed information of Node D (such as IP address, login credentials, etc.). The repair script can perform a series of standardized operations to fix the high CPU usage problem of Node D.

[0079] Step S503: If the faulty node is inaccessible, the automatic configuration file for node replacement in the automatic operation and maintenance tool is called to repair the faulty node.

[0080] For example, the event callback mechanism fails when trying to access Node E (another failed node), indicating that Node E may be completely down or the network connection is disconnected. In this case, the event callback mechanism will call another automation profile (the automation profile for node replacement), which is used to select a new node from the spare node list to replace Node E.

[0081] In summary, through the above steps, the automated operation and maintenance tool can select the most appropriate recovery strategy according to the type of faulty node, thereby improving the flexibility and efficiency of automated operation and maintenance.

[0082] In an optional embodiment, if Figure 6 As shown, if the faulty node is accessible, the automatic configuration file for node repair in the automatic operation and maintenance tool is called to repair the faulty node, including:

[0083] Step S601: Call the interface of the container orchestration system through the automated operation and maintenance tool to mark the faulty node as unschedulable.

[0084] Optionally, before starting the repair process, you first need to ensure that the faulty node does not receive new workloads. This can be achieved by marking the faulty node as "unschedulable", that is, modifying the state of the faulty node through the interface of the container orchestration system to prevent the scheduling of new containers. For example, the target alarm information generated is about the abnormal operation of Node F, and the event callback mechanism detects that the network connectivity of Node F is normal. Then the first step is to call the interface in the repair script to mark Node F as unschedulable. This can prevent new containers from being assigned to the node, thereby reducing the load on the node and creating conditions for subsequent repair operations.

[0085] Step S602: Deleting the deployed container from the failed node.

[0086] Optionally, to ensure that all applications and services on the current faulty node stop running, you can delete all deployed containers on the faulty node for repair operations. For example, after marking Node F as unschedulable, you can use the command to evict all running containers on the node to ensure that all workloads have been safely migrated to other healthy nodes in the cluster. This can avoid data loss or service interruption during the repair process.

[0087] Step S603: Add a target tag to prevent the faulty node from still generating an alarm during the repair process.

[0088] For example, to prevent an alarm from being generated again for Node F during the repair process, the repair script will add a target label, such as repair-in-progress=true, to the metadata of Node F. In this way, the alarm rule can be configured to ignore nodes with this label, avoiding duplicate alarms.

[0089] Step S604: Delete directories and files related to the container orchestration system in the failed node.

[0090] Optionally, during the fault repair process of the container orchestration system, configuration files and data related to the system need to be completely cleared to ensure that the repaired system has no remaining misconfigurations or damaged files. For example, during the repair process, the directories and files related to the container orchestration system on Node F need to be completely cleared to ensure that the components that are subsequently reinstalled can run normally.

[0091] Step S605: reinstall the container orchestration system component in the faulty node.

[0092] Optionally, reinstall all necessary container orchestration system components to restore the container orchestration system to a stable working state, ensuring that all components are running normally and the node can rejoin the cluster.

[0093] Step S606: re-add the failed node to the container orchestration system cluster and delete the target label.

[0094] For example, once the container orchestration system components on Node F are reinstalled, you can use the command to rejoin Node F to the container orchestration system cluster. At the same time, delete the target label of repair-in-progress=true so that the node can be monitored normally.

[0095] Step S607: Set the faulty node to a schedulable state through an automated operation and maintenance tool to complete repair of the faulty node.

[0096] For example, after Node F successfully joins the cluster, the last step is to mark Node F as schedulable through the repair script or directly using the command. This means that the node is ready to receive new containers and workloads, and the repair process is complete.

[0097] In summary, through the above technical solutions, the automated operation and maintenance tools can perform standardized repairs on network-reachable faulty nodes without manual intervention. This not only greatly improves the efficiency of fault recovery, but also reduces human errors, ensuring the stability and availability of the cluster.

[0098] In an optional embodiment, if Figure 7 As shown, if the faulty node is inaccessible, the automatic configuration file for node replacement in the automatic operation and maintenance tool is called to repair the faulty node, including:

[0099] Step S701: delete the faulty node from the container orchestration system.

[0100] Optionally, when a failed node becomes inaccessible at the network level, it means that the failed node may have encountered a hardware failure, a network problem, or a serious software crash. The automated operation and maintenance tool will call the structure of the container orchestration system to safely delete the failed node from the cluster.

[0101] Step S702: determine a standby node from a standby node pool of the container orchestration system through an event callback mechanism.

[0102] Optionally, after the failed node is deleted from the container orchestration system, a healthy node can be selected from the cluster's spare node pool as a replacement for the failed node. The above spare node pool is a concept used for high availability and fast fault recovery in the container orchestration system architecture. It refers to a preconfigured collection of multiple spare or standby nodes. These nodes do not participate in the daily workload processing of the cluster under normal circumstances. However, when a node failure occurs in the cluster or computing resources need to be expanded, the spare node can be quickly activated and joined to the cluster to replace the failed node or increase the computing power of the cluster.

[0103] For example, a node can be selected from a list of standby nodes based on predefined logic. This list can be stored in an asset definition file of an automated operation and maintenance tool, which contains information such as the IP address and login credentials of the standby node.

[0104] Step S703: Add a standby node to the cluster to replace the failed node, thereby completing the repair of the failed node.

[0105] Optionally, the selected standby node will be configured and added to the container orchestration system cluster to replace the function of the failed node and ensure that the cluster's resource pool and computing power remain stable.

[0106] In summary, through the above technical solutions, the automated operation and maintenance tool can maintain the stability and availability of the cluster through the replacement mechanism when the faulty node cannot be accessed. This method not only reduces the service interruption time, but also ensures that the cluster's resource allocation and scheduling can quickly return to normal.

[0107] In an optional embodiment, if Figure 8 As shown, when the event callback mechanism receives the target alarm information, after calling the automated operation and maintenance tool to repair the faulty node, the method further includes:

[0108] Step S801: Verify whether the components of the container orchestration system are successfully started at the faulty node by using an automated operation and maintenance tool.

[0109] Optionally, use an automated operation and maintenance tool to verify whether the components of the container orchestration system are successfully started at the faulty node. This process involves the use of remote connection and status check tools, and the specific process can be divided into the following steps: the automated operation and maintenance tool remotely connects to the repaired faulty node; after connecting, the automated operation and maintenance tool executes commands to check whether the core components of the container orchestration system are running; confirming whether the core components of the container orchestration system are operating normally can include checking the service log to see if there is any error information about startup failure or abnormal operation; using the management tool of the container orchestration system to verify the status, check the status of the node and the operation of the components; some components may provide health check endpoints, and HTTP requests can be used to check whether the components are responsive and in a healthy state; the automated operation and maintenance tool will record the status check results of the components and generate a repair report. If all components are confirmed to have started successfully and are running normally, the tool will record this status and can notify the operation and maintenance team through event logs, emails or other communication methods that the repair process has been completed.

[0110] Step S802: If the startup is successful, it is confirmed that the repair of the faulty node is completed.

[0111] For example, if the above inspection results indicate that the components of the container orchestration system have been successfully started, it can be confirmed that the repair process of the faulty node node-f has been successfully completed.

[0112] In summary, through the above technical solutions and this series of inspection and verification steps, the automated operation and maintenance tools can ensure that the faulty nodes can resume normal services after repair, providing a reliable, stable and predictable operating environment, thereby improving the stability and efficiency of the entire container orchestration system.

[0113] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0114] Example 2

[0115] The embodiment of the present application also provides a device for repairing a faulty node in a container orchestration system. It should be noted that the device for repairing a faulty node in a container orchestration system in the embodiment of the present application can be used to execute the method for repairing a faulty node in a container orchestration system provided in the embodiment of the present application. The following introduces the device for repairing a faulty node in a container orchestration system provided in the embodiment of the present application.

[0116] According to an embodiment of the present application, a device for implementing the above-mentioned method for repairing a failed node in the container orchestration system is also provided, such as Fig. 9As shown, the device includes: an acquisition unit 901, a first generation unit 902, a sending unit 903 and a first repairing unit 904.

[0117] Specifically, the acquisition unit 901 is used to obtain the monitoring indicator of the target node in the cluster by the container orchestration system, wherein the monitoring indicator is the measurement data used to describe the operation indicator of the target node, wherein the operation indicator is at least one of the following: resource usage, operation status and operation performance;

[0118] A first generating unit 902 is configured to generate first alarm information according to the monitoring indicator and the alarm rule when the target node is determined to be a faulty node according to the monitoring indicator and the predefined alarm rule;

[0119] The sending unit 903 is used to generate target alarm information according to the first alarm information through the alarm manager, and send the target alarm information to the event callback mechanism;

[0120] The first repairing unit 904 is used to call the automated operation and maintenance tool to repair the faulty node when the event callback mechanism receives the target alarm information.

[0121] The device for repairing a faulty node of a container orchestration system provided in an embodiment of the present application obtains monitoring indicators of a target node in a cluster of the container orchestration system through an acquisition unit 901, wherein the monitoring indicators are measurement data for describing the operating indicators of the target node, wherein the operating indicators are at least one of the following: resource usage, operating status, and operating performance; the first generation unit 902 generates a first alarm message according to the monitoring indicators and the alarm rules when the target node is determined to be a faulty node according to the monitoring indicators and predefined alarm rules; the sending unit 903 generates target alarm information according to the first alarm information through the alarm manager, and sends the target alarm information to the event callback mechanism; the first repair unit 904 calls an automated operation and maintenance tool to repair the faulty node when the event callback mechanism receives the target alarm information, thereby solving the technical problem of low repair efficiency due to reliance on manual repair when a node in a cluster of a container orchestration system fails in the related technology, thereby achieving the technical effect of improving the efficiency of repairing faulty nodes in a cluster of a container orchestration system.

[0122] Optionally, in the repair device for a faulty node of a container orchestration system provided in an embodiment of the present application, the device further includes: a deduplication unit, for receiving the first alarm information through the alarm manager before generating target alarm information based on the first alarm information through the alarm manager and sending the target alarm information to the event callback mechanism, and performing a deduplication operation on the first alarm information according to a predefined deduplication rule to obtain second alarm information; a grouping unit, for grouping the alarm information in the second alarm information according to a grouping strategy; and a conversion unit, for converting the format of the grouped second alarm information to obtain third alarm information.

[0123] Optionally, in the repair device for a faulty node of a container orchestration system provided in an embodiment of the present application, the device also includes: an evaluation unit, which is used to perform a priority evaluation on multiple alarm information in the third alarm information through the alarm manager before generating target alarm information based on the first alarm information and sending the target alarm information to the event callback mechanism, and assign a priority to each alarm information in the third alarm information according to a preset priority rule; and a second generation unit, which is used to generate the target alarm information based on the alarm information with the highest priority in the third alarm information.

[0124] Optionally, in the device for repairing a faulty node in a container orchestration system provided in an embodiment of the present application, the device further includes: a detection unit, which is used to detect the network connectivity of the faulty node through the event callback mechanism before calling the automated operation and maintenance tool to repair the faulty node when the event callback mechanism receives the target alarm information, so as to determine whether the faulty node can be accessed through the network connection; a second repair unit, which is used to call an automated configuration file for node repair in the automated operation and maintenance tool to repair the faulty node if the faulty node is accessible; and a third repair unit, which is used to call an automated configuration file for node replacement in the automated operation and maintenance tool to repair the faulty node if the faulty node is not accessible.

[0125] Optionally, in the repair device for a faulty node in a container orchestration system provided in an embodiment of the present application, the second repair unit includes: a calling module, used to call the interface of the container orchestration system through an automated operation and maintenance tool to mark the faulty node as an unschedulable state; a first deletion module, used to delete the deployed container from the faulty node; a first addition module, used to add a target label to prevent the faulty node from still generating an alarm during the repair process; a second deletion module, used to delete directories and files related to the container orchestration system in the faulty node; an installation module, used to reinstall the container orchestration system components in the faulty node; a third deletion module, used to rejoin the faulty node to the container orchestration system cluster and delete the target label; and a repair module, used to set the faulty node to a schedulable state through an automated operation and maintenance tool to complete the repair of the faulty node.

[0126] Optionally, in the device for repairing a faulty node in a container orchestration system provided in an embodiment of the present application, the third repair unit includes: a fourth deletion module, used to delete the faulty node from the container orchestration system; a determination module, used to determine a backup node from a backup node pool of the container orchestration system through an event callback mechanism; and a second addition module, used to add the backup node to the cluster to replace the faulty node and complete the repair of the faulty node.

[0127] Optionally, in the device for repairing a faulty node of a container orchestration system provided in an embodiment of the present application, the device further includes: a verification unit, which is used to call an automated operation and maintenance tool to repair the faulty node when the event callback mechanism receives the target alarm information, and then verify through the automated operation and maintenance tool whether the components of the container orchestration system are successfully started at the faulty node; and a confirmation unit, which is used to confirm that the repair of the faulty node is completed if it is successfully started.

[0128] It should be noted that the acquisition unit 901, the first generation unit 902, the sending unit 903 and the first repair unit 904 correspond to steps S201 to S204 in Example 1, and the four units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned embodiment 1. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules can also be part of the device and can be run in the computer terminal 10 provided in the first embodiment.

[0129] Example 3

[0130] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal or an electronic device.

[0131] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0132] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the method for repairing a faulty node in a container orchestration system: obtaining monitoring indicators of the container orchestration system for a target node in a cluster, wherein the monitoring indicators are measurement data used to describe the operating indicators of the target node, wherein the operating indicators are at least one of the following: resource usage, operating status, and operating performance; when the target node is determined to be a faulty node based on the monitoring indicators and predefined alarm rules, generating a first alarm message based on the monitoring indicators and the alarm rules; generating target alarm information based on the first alarm information through an alarm manager, and sending the target alarm information to an event callback mechanism; when the event callback mechanism receives the target alarm information, calling an automated operation and maintenance tool to repair the faulty node.

[0133] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the method for repairing a faulty node in a container orchestration system: before generating target alarm information based on the first alarm information through the alarm manager and sending the target alarm information to the event callback mechanism, the method also includes: receiving the first alarm information through the alarm manager, performing a deduplication operation on the first alarm information according to a predefined deduplication rule to obtain second alarm information; grouping the alarm information in the second alarm information according to a grouping strategy; and converting the format of the grouped second alarm information to obtain third alarm information.

[0134] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the method for repairing a faulty node in a container orchestration system: before generating target alarm information based on the first alarm information through the alarm manager and sending the target alarm information to the event callback mechanism, the method also includes: performing priority evaluation on multiple alarm information in the third alarm information through the alarm manager, and assigning a priority to each alarm information in the third alarm information according to a preset priority rule; generating target alarm information based on the alarm information with the highest priority in the third alarm information.

[0135] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the method for repairing a faulty node in a container orchestration system: when the event callback mechanism receives the target alarm information, before calling the automated operation and maintenance tool to repair the faulty node, the method also includes: detecting the network connectivity of the faulty node through the event callback mechanism to determine whether the faulty node can be accessed through the network connection; if the faulty node can be accessed, calling the automated configuration file for node repair in the automated operation and maintenance tool to repair the faulty node; if the faulty node is not accessible, calling the automated configuration file for node replacement in the automated operation and maintenance tool to repair the faulty node.

[0136] Optionally, the computer terminal may execute the following program codes of the method for repairing a faulty node in a container orchestration system: if the faulty node is accessible, the automated configuration file for node repair in the automated operation and maintenance tool is called to repair the faulty node, including: calling the interface of the container orchestration system through the automated operation and maintenance tool to mark the faulty node as an unschedulable state; deleting the deployed container from the faulty node; adding a target label to prevent the faulty node from still generating an alarm during the repair process; deleting directories and files related to the container orchestration system in the faulty node; reinstalling the container orchestration system components in the faulty node; rejoining the faulty node to the container orchestration system cluster and deleting the target label; setting the faulty node to a schedulable state through the automated operation and maintenance tool to complete the repair of the faulty node.

[0137] Optionally, the computer terminal may execute the following program codes of the method for repairing a faulty node in a container orchestration system: if the faulty node is inaccessible, calling an automated configuration file for node replacement in an automated operation and maintenance tool to repair the faulty node, including: deleting the faulty node from the container orchestration system; determining a spare node from a spare node pool of the container orchestration system through an event callback mechanism; adding the spare node to the cluster to replace the faulty node, thereby completing the repair of the faulty node.

[0138] Optionally, the computer terminal may execute the following program codes of the method for repairing a faulty node in a container orchestration system: when the event callback mechanism receives the target alarm information, after calling the automated operation and maintenance tool to repair the faulty node, the method further includes: verifying whether the components of the container orchestration system are successfully started at the faulty node through the automated operation and maintenance tool; if successfully started, confirming that the repair of the faulty node is complete.

[0139] Optionally, Fig.10 is a structural block diagram of an electronic device according to an embodiment of the present application. Fig.10 As shown, the electronic device may include: one or more ( Fig.10 (only one is shown) processor 1002, memory 1004, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0140] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for repairing a faulty node in a container orchestration system in an embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the method for repairing a faulty node in a container orchestration system described above is implemented. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely disposed relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0141] The processor may call the information and application program stored in the memory through the transmission device to execute the above steps in the method for repairing a faulty node of the container orchestration system.

[0142] By adopting the embodiment of the present application, a solution for repairing a faulty node in a container orchestration system is provided. By obtaining the monitoring indicators of the target node in the cluster by the container orchestration system, wherein the monitoring indicators are measurement data used to describe the operating indicators of the target node, wherein the operating indicators are at least one of the following: resource usage, operating status, and operating performance; when the target node is determined to be a faulty node based on the monitoring indicators and predefined alarm rules, a first alarm message is generated based on the monitoring indicators and the alarm rules; a target alarm message is generated based on the first alarm message by the alarm manager, and the target alarm message is sent to the event callback mechanism; when the event callback mechanism receives the target alarm message, an automated operation and maintenance tool is called to repair the faulty node, thereby solving the technical problem of relying on manual repair and low repair efficiency when a node in the cluster of the container orchestration system fails in the related technology. In the present application, by obtaining the monitoring indicators of the target node in the cluster of the container orchestration system, if the target node is determined to be a faulty node based on the monitoring indicators and predefined alarm rules, a first alarm message is generated, and the alarm manager generates target alarm information based on the first alarm information and sends it to the event callback mechanism. After receiving the target alarm information, the event callback mechanism calls the automated operation and maintenance tool to repair the faulty node, thereby achieving the technical effect of improving the efficiency of repairing faulty nodes in the container orchestration system cluster.

[0143] It can be understood by those skilled in the art that Fig.10 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.10The structure of the electronic device is not limited. Fig.10 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.10 Different configurations are shown.

[0144] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0145] Example 4

[0146] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the method for repairing a failed node in a container orchestration system provided in the first embodiment.

[0147] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0148] Optionally, in this embodiment, the storage medium is configured to store a program code for executing a method for repairing a failed node in the container orchestration system.

[0149] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of a method for repairing a failed node in a container orchestration system.

[0150] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0151] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0153] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly or all or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0156] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for repairing a faulty node in a container orchestration system, characterized in that: include: Obtaining monitoring indicators of the target node in the cluster by the container orchestration system, wherein the monitoring indicators are measurement data used to describe the operating indicators of the target node, wherein the operating indicators are at least one of the following: resource usage, operating status, and operating performance; In the case where the target node is determined to be a faulty node according to the monitoring indicator and the predefined alarm rule, generating first alarm information according to the monitoring indicator and the alarm rule; Generate target alarm information according to the first alarm information through the alarm manager, and send the target alarm information to the event callback mechanism; When the event callback mechanism receives the target alarm information, an automated operation and maintenance tool is called to repair the faulty node.

2. The method according to claim 1, characterized in that Before generating target alarm information according to the first alarm information through the alarm manager and sending the target alarm information to the event callback mechanism, the method further includes: receiving the first alarm information through the alarm manager, and performing a deduplication operation on the first alarm information according to a predefined deduplication rule to obtain second alarm information; Grouping the alarm information in the second alarm information according to a grouping strategy; The format of the grouped second alarm information is converted to obtain third alarm information.

3. The method according to claim 2, characterized in that Before generating target alarm information according to the first alarm information through the alarm manager and sending the target alarm information to the event callback mechanism, the method further includes: Performing priority evaluation on multiple alarm information in the third alarm information through the alarm manager, and assigning a priority to each alarm information in the third alarm information according to a preset priority rule; Generate target alarm information according to the alarm information with the highest priority in the third alarm information.

4. The method according to claim 1, characterized in that: When the event callback mechanism receives the target alarm information, before calling the automated operation and maintenance tool to repair the faulty node, the method further includes: The network connectivity of the faulty node is detected by the event callback mechanism to determine whether the faulty node can be accessed through the network connection; If the faulty node is accessible, the automated configuration file for node repair in the automated operation and maintenance tool is called to repair the faulty node; If the faulty node is inaccessible, the automatic configuration file for node replacement in the automatic operation and maintenance tool is called to repair the faulty node.

5. The method according to claim 3, characterized in that: If the faulty node is accessible, the automatic configuration file for node repair in the automatic operation and maintenance tool is called to repair the faulty node, including: Calling the interface of the container orchestration system through the automated operation and maintenance tool to mark the faulty node as unschedulable; Deleting the deployed container from the failed node; Add a target tag to prevent the faulty node from still generating an alarm during the repair process; Deleting directories and files related to the container orchestration system in the faulty node; Reinstalling the container orchestration system component in the failed node; Rejoining the failed node to the container orchestration system cluster and deleting the target label; The faulty node is set to a schedulable state through the automated operation and maintenance tool to complete the repair of the faulty node.

6. The method according to claim 3, characterized in that If the faulty node is inaccessible, the automatic configuration file for node replacement in the automatic operation and maintenance tool is called to repair the faulty node, including: Deleting the faulty node from the container orchestration system; Determine a standby node from a standby node pool of the container orchestration system through the event callback mechanism; The standby node is added to the cluster to replace the failed node, thereby completing the repair of the failed node.

7. The method according to claim 1, characterized in that When the event callback mechanism receives the target alarm information, after calling the automated operation and maintenance tool to repair the faulty node, the method further includes: Verify, by the automated operation and maintenance tool, at the fault node whether the components of the container orchestration system are successfully started; If the startup is successful, it is confirmed that the repair of the faulty node is completed.

8. A device for repairing a failed node in a container orchestration system, characterized in that: include: An acquisition unit, configured to acquire monitoring indicators of a target node in a cluster by a container orchestration system, wherein the monitoring indicators are measurement data used to describe an operating indicator of the target node, wherein the operating indicator is at least one of the following: resource usage, operating status, and operating performance; A first generating unit, configured to generate first alarm information according to the monitoring indicator and the alarm rule when the target node is determined to be a faulty node according to the monitoring indicator and the predefined alarm rule; A sending unit, configured to generate target alarm information according to the first alarm information through an alarm manager, and send the target alarm information to an event callback mechanism; The first repairing unit is used to call the automated operation and maintenance tool to repair the faulty node when the event callback mechanism receives the target alarm information.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the method for repairing a failed node in a container orchestration system according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program, when running, executes the method for repairing a failed node in a container orchestration system as described in any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by the processor, the steps of the method for repairing a failed node in a container orchestration system according to any one of claims 1 to 7 are implemented.

Images