Cluster fault processing method, electronic equipment, storage medium and program product
By acquiring fault information from the ClickHouse cluster and matching it with a self-healing strategy, faults are automatically recovered, solving the problem of low efficiency in manual recovery and achieving efficient fault recovery and improved system reliability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-24
- Publication Date
- 2026-04-03
AI Technical Summary
In ClickHouse clusters, fault recovery relies on manual operations, resulting in low fault recovery efficiency.
By acquiring fault type and status information of the target cluster, a self-healing strategy is matched to automatically recover from the fault, including restarting services or operating systems, or triggering manual operations. The self-healing strategy can be upgraded to improve the success rate by accurately matching the fault type and cluster status.
It improves the success rate and efficiency of fault recovery, reduces operation and maintenance costs, enhances the availability and reliability of the management system, and reduces human error.
Smart Images
Figure CN121792306A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data technology, and in particular to a cluster fault handling method, electronic device, storage medium and program product. Background Technology
[0002] ClickHouse is a high-performance, column-oriented, structured query language (SQL) database management system (DBMS) for online big data analysis and processing.
[0003] Enterprise-level ClickHouse deployments are typically clustered, containing multiple ClickHouse nodes as storage and computing units, multiple gateway nodes for gateway integration and forwarding, and multiple installation and operation (ZooKeeper) nodes for distributed coordination and management. As such, the ClickHouse cluster architecture is quite complex. As the number of nodes in the cluster increases with the scale of business, the probability of single-point service failures during cluster operation also increases. In related technologies, fault recovery mainly relies on manual operations by maintenance personnel. However, manual fault recovery is slow, resulting in low fault recovery efficiency.
[0004] Therefore, improving fault recovery efficiency is a technical problem that urgently needs to be solved. Summary of the Invention
[0005] This application provides a cluster fault handling method, electronic device, storage medium, and program product to improve fault recovery efficiency.
[0006] In a first aspect, embodiments of this application provide a cluster fault handling method, including:
[0007] Obtain the fault type information and cluster status information of the target node in the target cluster. The fault type information of the target node indicates whether the fault type of the target node is a service fault or a host fault. The cluster status information includes the number of currently faulty nodes and similar fault information. The similar fault information indicates whether the time of the fault occurrence of the target node is within the first fault recovery window. The first fault recovery window is the fault recovery window corresponding to the first fault that has self-healed. The fault of the target node is the same as or similar to the first fault.
[0008] Perform a matching operation, which includes: matching a first self-healing strategy for the fault of the target node based on the fault type information of the target node and the cluster status information;
[0009] The first self-healing strategy is executed to recover the target node from failure.
[0010] In one possible implementation, the matching operation includes any of the following:
[0011] If the fault type information of the target node indicates that the fault type of the target node is the service fault, and the number of the current fault nodes is less than or equal to the threshold of self-healing fault nodes, and the similar fault information indicates that the fault occurrence time of the target node is outside the first fault recovery window, then restarting the service will be matched as the first self-healing strategy.
[0012] If the fault type information of the target node indicates that the fault type of the target node is the host fault, and the number of currently faulty nodes is less than or equal to the threshold of self-healing faulty nodes, and the similar fault information indicates that the time of the fault occurrence of the target node is outside the first fault recovery window, then restarting the operating system will be matched as the first self-healing strategy.
[0013] In one possible implementation, the matching operation includes:
[0014] If the number of currently faulty nodes is greater than the threshold of self-healing faulty nodes, or if the similar fault information indicates that the fault occurrence time of the target node is within the first fault recovery window, then manual operation will be triggered to match the first self-healing strategy.
[0015] In one possible implementation, the method further includes:
[0016] If the first self-healing strategy fails, a switching operation is performed. The switching operation includes: switching the fault self-healing strategy of the target node to a second self-healing strategy according to a preset strategy upgrade path. The level of the second self-healing strategy is higher than that of the first self-healing strategy.
[0017] Repeat the matching and switching operations until the fault of the target node is recovered or manual operation is triggered.
[0018] In one possible implementation, if the first self-healing strategy fails, a switching operation is performed, including:
[0019] If the first self-healing strategy fails to execute for the first time, the first self-healing strategy is executed repeatedly until the number of times the first self-healing strategy is executed reaches the repeated execution threshold and the first self-healing strategy fails to execute, then the switching operation is executed.
[0020] In one possible implementation, the target cluster includes multiple nodes, each node deploying a data acquisition service and specific services;
[0021] The step of obtaining the fault type information of the target node in the target cluster includes:
[0022] For any node in the target cluster, obtain the health information of a specific service in the node within a self-healing window and the status information of the data collection service. The self-healing window includes multiple collection times, the health information of the specific service within the self-healing window includes the health information corresponding to the multiple collection times, and the status information of the data collection service indicates whether the data collection service is healthy.
[0023] Based on the health information of the specific service within the self-healing window and / or the status information of the data acquisition service, a diagnostic command is executed, which is used to diagnose the health status of the specific service in real time.
[0024] If the diagnostic command cannot be executed, the fault type of the node is determined to be a host fault;
[0025] If the diagnostic command is successfully executed and the execution result indicates that the health status of the specific service is unhealthy, then the fault type of the node is determined to be a service fault.
[0026] In one possible implementation, executing the diagnostic command based on the health information of the specific service within the self-healing window and / or the status information of the data acquisition service includes any one of the following:
[0027] If the health information of the specific service within the self-healing window is incomplete, and the status information of the data collection service indicates that the data collection service is unhealthy, then the data collection service is restarted and the diagnostic command is executed.
[0028] If the health information of the specific service within the self-healing window is incomplete, and the status information of the data collection service indicates that the data collection service is healthy, then the diagnostic command is executed.
[0029] If the health information of the specific service is complete within the self-healing window, and the health information corresponding to at least one of the multiple collection times indicates that the specific service is unhealthy, then the diagnostic command is executed.
[0030] In one possible implementation, the method further includes:
[0031] Based on the services applicable to the target cluster, determine the self-healing parameters corresponding to the target cluster. The self-healing parameters include at least one of the following: the duration of the self-healing window, the duration of the fault recovery window, the threshold of self-healable fault nodes, the percentage threshold of self-healable fault nodes in the total number of nodes in the cluster, and the repeated execution threshold.
[0032] Secondly, embodiments of this application provide a cluster fault handling apparatus, comprising:
[0033] The acquisition module is used to acquire fault type information and cluster status information of target nodes in the target cluster. The fault type information of the target node indicates that the fault type of the target node is a service fault or a host fault. The cluster status information includes the number of currently faulty nodes and similar fault information. The similar fault information indicates whether the time of occurrence of the fault of the target node is within the first fault recovery window. The first fault recovery window is the fault recovery window corresponding to the first fault that has self-healed. The fault of the target node is the same as or similar to the first fault.
[0034] A matching module is used to perform a matching operation, the matching operation including: matching a first self-healing strategy for the fault of the target node based on the fault type information of the target node and the cluster status information;
[0035] The fault recovery module is used to execute the first self-healing strategy to recover the fault of the target node.
[0036] In one possible implementation, the matching module is specifically configured to perform any of the following:
[0037] If the fault type information of the target node indicates that the fault type of the target node is the service fault, and the number of the current fault nodes is less than or equal to the threshold of self-healing fault nodes, and the similar fault information indicates that the fault occurrence time of the target node is outside the first fault recovery window, then restarting the service will be matched as the first self-healing strategy.
[0038] If the fault type information of the target node indicates that the fault type of the target node is the host fault, and the number of currently faulty nodes is less than or equal to the threshold of self-healing faulty nodes, and the similar fault information indicates that the time of the fault occurrence of the target node is outside the first fault recovery window, then restarting the operating system will be matched as the first self-healing strategy.
[0039] In one possible implementation, the matching module is specifically used for:
[0040] If the number of currently faulty nodes is greater than the threshold of self-healing faulty nodes, or if the similar fault information indicates that the fault occurrence time of the target node is within the first fault recovery window, then manual operation will be triggered to match the first self-healing strategy.
[0041] In one possible implementation, the device further includes a switching module and a repetition module, wherein,
[0042] The switching module is used to perform a switching operation if the first self-healing strategy fails. The switching operation includes: switching the fault self-healing strategy of the target node to a second self-healing strategy according to a preset strategy upgrade path. The level of the second self-healing strategy is higher than that of the first self-healing strategy.
[0043] The repeat execution module is used to repeatedly execute the matching operation and the switching operation until the fault of the target node is recovered or manual operation is triggered.
[0044] In one possible implementation, the switching module is specifically used for:
[0045] If the first self-healing strategy fails to execute for the first time, the first self-healing strategy is executed repeatedly until the number of times the first self-healing strategy is executed reaches the repeated execution threshold and the first self-healing strategy fails to execute, then the switching operation is executed.
[0046] In one possible implementation, the target cluster includes multiple nodes, each node deploying a data acquisition service and a specific service; the device further includes a diagnostic module and a determination module, wherein,
[0047] The acquisition module is further configured to acquire, for any node in the target cluster, the health information of a specific service within a self-healing window and the status information of the data acquisition service within a self-healing window, wherein the self-healing window includes multiple acquisition times, the health information of the specific service within the self-healing window includes the health information corresponding to the multiple acquisition times, and the status information of the data acquisition service indicates whether the data acquisition service is healthy.
[0048] The diagnostic module is used to execute diagnostic commands based on the health information of the specific service within the self-healing window and / or the status information of the data acquisition service. The diagnostic commands are used to diagnose the health status of the specific service in real time.
[0049] The determining module is configured to determine the fault type of the node as a host fault if the diagnostic command cannot be executed; and to determine the fault type of the node as a service fault if the diagnostic command is successfully executed and the execution result indicates that the health status of the specific service is unhealthy.
[0050] In one possible implementation, the diagnostic module is specifically configured to perform any of the following:
[0051] If the health information of the specific service within the self-healing window is incomplete, and the status information of the data collection service indicates that the data collection service is unhealthy, then the data collection service is restarted and the diagnostic command is executed.
[0052] If the health information of the specific service within the self-healing window is incomplete, and the status information of the data collection service indicates that the data collection service is healthy, then the diagnostic command is executed.
[0053] If the health information of the specific service is complete within the self-healing window, and the health information corresponding to at least one of the multiple collection times indicates that the specific service is unhealthy, then the diagnostic command is executed.
[0054] In one possible implementation, the determining module is further configured to:
[0055] Based on the services applicable to the target cluster, determine the self-healing parameters corresponding to the target cluster. The self-healing parameters include at least one of the following: the duration of the self-healing window, the duration of the fault recovery window, the threshold of self-healable fault nodes, the percentage threshold of self-healable fault nodes in the total number of nodes in the cluster, and the repeated execution threshold.
[0056] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;
[0057] The memory stores computer-executed instructions;
[0058] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.
[0059] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.
[0060] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.
[0061] This application provides a cluster fault handling method, electronic device, storage medium, and program product. Based on the fault type information and cluster status information of the target node in the target cluster, a first self-healing strategy is matched for the fault of the target node. This achieves accurate matching of the self-healing strategy, avoids invalid operations, and thus improves the success rate and efficiency of fault recovery. Attached Figure Description
[0062] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0063] Figure 1This is a schematic diagram of an application scenario provided by an embodiment of this application;
[0064] Figure 2 A flowchart illustrating a cluster fault handling method provided in this application embodiment. Figure 1 ;
[0065] Figure 3 A flowchart illustrating a cluster fault handling method provided in this application embodiment. Figure 2 ;
[0066] Figure 4 A flowchart illustrating a cluster fault handling method provided in this application embodiment. Figure 3 ;
[0067] Figure 5 This is a schematic diagram of the structure of a cluster fault handling device provided in an embodiment of this application;
[0068] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0069] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0070] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0071] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0072] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.
[0073] It should be understood that the terms "comprising" or "including" indicate the presence of the aforementioned features, steps, or operations, but do not preclude the presence, occurrence, or addition of one or more other features, steps, or operations.
[0074] The terms "first," "second," etc., used in the embodiments of this application are for illustrative purposes and to distinguish the objects being described. They do not indicate any order and do not imply any special limitation on the number of objects in the embodiments of this application. They do not constitute any limitation on the embodiments of this application.
[0075] In this application, terms such as “exemplary,” “in some embodiments,” “for example,” and “in other embodiments” are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as “exemplary” or “for example” in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the term “exemplary” is used to present the concept in a specific manner.
[0076] It should be noted that the cluster fault handling method, apparatus, device and storage medium of this application can be used in the field of big data technology, or in any field other than big data technology. The application field of the cluster fault handling method, apparatus, device and storage medium of this application is not limited.
[0077] Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. Please refer to [link / reference]. Figure 1 It includes a management system 101 and multiple clusters 102. The management system 101, which can also be called a management platform, can be used to manage the service failure self-healing process of multiple clusters.
[0078] Figure 1 This is merely an illustrative diagram of one application scenario and does not constitute a limitation on the application scenarios of the technical solutions provided in this application. For example, the number of clusters 102 can also be other values.
[0079] To address the technical problems mentioned above, this application provides a cluster fault handling method. Based on the fault type information and cluster status information of the target node in the target cluster, a first self-healing strategy is matched for the fault of the target node, achieving accurate matching of the self-healing strategy, avoiding invalid operations, and thus improving the success rate and efficiency of fault recovery.
[0080] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0081] Figure 2 A flowchart illustrating a cluster fault handling method provided in this application embodiment. Figure 1 ,like Figure 2 As shown, the method includes:
[0082] S201. Obtain the fault type information and cluster status information of the target node in the target cluster. The fault type information of the target node indicates whether the fault type of the target node is a service fault or a host fault. The cluster status information includes the number of currently faulty nodes and similar fault information. The similar fault information indicates whether the time of the fault occurrence of the target node is within the first fault recovery window. The first fault recovery window is the fault recovery window corresponding to the first fault that has self-healed. The fault of the target node is the same as or similar to the first fault.
[0083] The execution entity in this application embodiment can be an electronic device (such as a server) or a cluster fault handling device installed in an electronic device. The cluster fault handling device can be implemented by software or by a combination of software and hardware. The electronic device is the physical device that carries the management system.
[0084] For any cluster managed by the management system, the cluster can include three types of nodes: clickhouse nodes, gateway nodes, and installation and running (zookeeper) nodes.
[0085] The target cluster can be a faulty cluster (hereinafter referred to as a faulty cluster). Among the multiple clusters managed by the management system, at least one is a faulty cluster, and the target cluster can be any one of the at least one faulty cluster. For example, the cluster in this application embodiment can be a ClickHouse cluster.
[0086] The method described in this application can be used to handle any single faulty cluster. The management system can handle multiple faulty clusters simultaneously or sequentially.
[0087] The target node can be a faulty node (hereinafter referred to as a faulty node), meaning the target node can be any one of at least one faulty node in the target cluster. For example, the target node can be a ClickHouse node, a gateway node, or an installed and running node.
[0088] For any given fault, once the fault has successfully self-healed, the corresponding fault recovery window can be opened to monitor whether a fault of the same or similar nature has occurred within the window. If a fault of the same or similar nature has occurred, a self-healing strategy different from the one corresponding to the fault will be used to recover the same or similar fault, thereby improving the stability after self-healing.
[0089] The start time of the fault recovery window can be the same as or later than the time when the corresponding fault self-healing is successful.
[0090] For example, the duration of a fault recovery window can be 20 minutes, and this application does not impose any restrictions on this. The duration of each fault recovery window can be configured according to actual conditions (such as at least one of the following: the business applicable to the cluster, the service corresponding to the fault recovery window, and the node corresponding to the fault recovery window). That is, the durations of different fault recovery windows can be the same or different.
[0091] The fault type information and cluster status information of the target node in the target cluster can be obtained in the following way: obtain relevant information about at least one fault to be processed in the target cluster, which may include the fault type information and cluster status information of the target node.
[0092] S202. Perform a matching operation, which includes: matching a first self-healing strategy for the target node's fault based on the target node's fault type information and cluster status information.
[0093] In one possible implementation, the matching operation may include any of the following:
[0094] (1) If the fault type information of the target node indicates that the fault type of the target node is a service fault, and the number of current fault nodes is less than or equal to the threshold of self-healing fault nodes, and the similar fault information indicates that the fault of the target node occurs outside the first fault recovery window, then restarting the service will be matched as the first self-healing strategy.
[0095] The self-healing strategy of restarting services is based on the components within a node, restarting the relevant services of the faulty node (i.e., the services that failed in the faulty node). The default upgradeable self-healing strategy can be restarting the operating system.
[0096] (2) If the fault type information of the target node indicates that the fault type of the target node is host fault, and the number of current fault nodes is less than or equal to the threshold of self-healing fault nodes, and the similar fault information indicates that the fault of the target node occurs outside the first fault recovery window, then restarting the operating system will be matched as the first self-healing strategy.
[0097] The self-healing strategy of restarting the operating system is based on nodes. It restarts the operating system of the faulty node. The default upgradeable self-healing strategy can be to trigger manual operation.
[0098] (3) If the number of current faulty nodes is greater than the threshold of self-healing faulty nodes or similar fault information indicates that the fault of the target node occurs within the first fault recovery window, then manual operation will be triggered to match the first self-healing strategy.
[0099] If the cluster fails to recover automatically or self-healing fails, the strategy of triggering manual operation can be activated. This strategy will automatically send or display a notification and retain the fault information to prompt the operation and maintenance personnel to handle the fault.
[0100] For example, the threshold for self-healing faulty nodes can be 16, but this application does not impose any restrictions on this. The threshold for self-healing faulty nodes can be the same or different for different clusters.
[0101] In addition to using the self-healing fault node threshold to determine whether the number of current faulty nodes exceeds the threshold, it can also be determined by using the percentage threshold of self-healing faulty nodes out of the total number of nodes in the cluster. That is, "the percentage of the current number of faulty nodes out of the total number of nodes in the cluster is less than or equal to the percentage threshold of self-healing faulty nodes out of the total number of nodes in the cluster" can be used instead of "the current number of faulty nodes is less than or equal to the self-healing faulty node threshold"; and "the percentage of the current number of faulty nodes out of the total number of nodes in the cluster is greater than the percentage threshold of self-healing faulty nodes out of the total number of nodes in the cluster" can be used instead of "the current number of faulty nodes is greater than the self-healing faulty node threshold".
[0102] For example, the percentage threshold mentioned above can be 50%, and this application does not impose any restrictions on it. The percentage thresholds mentioned above can be the same or different for different clusters.
[0103] S203. Execute the first self-healing strategy to recover from the failure of the target node.
[0104] Ideally, the first self-healing strategy should be executed once to recover the target node from failure. If the first self-healing strategy fails to execute on the first attempt, it can be executed repeatedly to reduce recovery interruptions caused by accidental failures and improve the robustness and success rate of fault self-healing.
[0105] In one possible implementation, it is necessary to limit the number of times the first self-healing strategy is repeatedly executed (i.e., set a threshold for the repeated execution of the first self-healing strategy) to avoid resource waste caused by the unlimited repeated execution of the first self-healing strategy.
[0106] If the number of repetitions of the first self-healing strategy reaches the repetition threshold and the first self-healing strategy fails, a higher-level self-healing strategy can be executed.
[0107] exist Figure 2 In the illustrated embodiment, a first self-healing strategy is matched to the fault of the target node based on the fault type information and cluster status information of the target node in the target cluster. This achieves accurate matching of the self-healing strategy, avoids invalid operations, and thus improves the success rate and efficiency of fault recovery.
[0108] exist Figure 2 Based on the illustrated embodiment, the solution after the first self-healing strategy fails is described in detail below.
[0109] Figure 3A flowchart illustrating a cluster fault handling method provided in this application embodiment. Figure 2 ,like Figure 3 As shown, the method includes:
[0110] S301. Obtain the fault type information and cluster status information of the target node in the target cluster. The fault type information of the target node indicates whether the fault type of the target node is a service fault or a host fault. The cluster status information includes the number of currently faulty nodes and similar fault information. The similar fault information indicates whether the time of the fault occurrence of the target node is within the first fault recovery window. The first fault recovery window is the fault recovery window corresponding to the first fault that has self-healed. The fault of the target node is the same as or similar to the first fault.
[0111] S302. Perform a matching operation, which includes: matching a first self-healing strategy for the target node's fault based on the target node's fault type information and cluster status information.
[0112] It should be noted that the execution process of S301 to S302 can be referred to the execution process of S201 to S202, and will not be repeated here.
[0113] S303, Implement the first self-healing strategy.
[0114] S304. If the first self-healing strategy fails, a switching operation is performed. The switching operation includes switching the fault self-healing strategy of the target node to the second self-healing strategy according to the preset strategy upgrade path. The level of the second self-healing strategy is higher than that of the first self-healing strategy.
[0115] If the first self-healing strategy is executed successfully, it means that the fault of the target node has been successfully eliminated.
[0116] S305. Repeat the matching and switching operations until the target node's fault is recovered or manual operation is triggered.
[0117] After a self-healing strategy is upgraded to triggering manual operation, the fault handling status can be recorded as abandoned. After any self-healing strategy other than triggering manual operation is executed successfully, the fault handling status can be recorded as successful. During the execution of any self-healing strategy other than triggering manual operation, the fault handling status can be recorded as processing. For faults that have not executed a self-healing strategy, the fault handling status can be recorded as unprocessed or pending.
[0118] For example, assuming the preset policy upgrade path is restart service → restart operating system → trigger manual operation, if the first self-healing policy is to restart service, then restart service will be executed. If restart service is successful, it means that the fault of the target node has been recovered. If restart service fails, then the fault self-healing policy will be switched to restart operating system and executed. If restart operating system is successful, it means that the fault of the target node has been recovered. If restart operating system fails, then the fault self-healing policy will be switched to trigger manual operation.
[0119] In one possible implementation, if the first self-healing strategy fails to execute on the first attempt, it is executed repeatedly until the number of times the first self-healing strategy is executed reaches the repeated execution threshold and the first self-healing strategy fails again, at which point a switching operation is performed. Similarly, for any self-healing strategy other than one that triggers manual operation, if the self-healing strategy fails to execute on the first attempt, it is executed repeatedly until the number of times the self-healing strategy is executed reaches the repeated execution threshold and the self-healing strategy fails again, at which point a switching operation is performed to switch to a higher-level self-healing strategy.
[0120] For example, assuming the preset policy upgrade path is restart service → restart operating system → trigger manual operation, if the first self-healing policy is to restart service, then restart service will be executed. If the restart service is successful, it means that the fault of the target node has been recovered. If the restart service fails on the first execution, then restart service will be executed repeatedly until restart service is successful or the number of repeated executions of restart service reaches the repeated execution threshold and restart service fails. Then the fault self-healing policy will be switched to restart operating system and executed. If restart operating system is successful, it means that the fault of the target node has been recovered. If restart operating system fails on the first execution, then restart operating system will be executed repeatedly until restart operating system is successful or the number of repeated executions of restart operating system reaches the repeated execution threshold and restart operating system fails. Then the fault self-healing policy will be switched to trigger manual operation.
[0121] Adding a repetitive execution mechanism can reduce recovery interruptions caused by accidental failures and improve the robustness and success rate of fault self-healing.
[0122] exist Figure 3 In the illustrated embodiment, by upgrading the self-healing strategy, the success rate of fault recovery can be significantly improved.
[0123] The above process details how to handle faults. The following section will combine... Figure 4 This section details how to differentiate between different faults before addressing a common problem.
[0124] Figure 4 A flowchart illustrating a cluster fault handling method provided in this application embodiment. Figure 3 ,like Figure 4As shown, the method includes:
[0125] S401. For any node in the target cluster, obtain the health information of a specific service within the self-healing window and the status information of the data acquisition service. The self-healing window includes multiple acquisition times. The health information of the specific service within the self-healing window includes the health information corresponding to multiple acquisition times. The status information of the data acquisition service indicates whether the data acquisition service is healthy.
[0126] For any cluster managed by the management system, the cluster can include three types of nodes: ClickHouse nodes, gateway nodes, and installation / running nodes. Each node in the cluster deploys a data acquisition service and a specific service. The specific service refers to the service installed on that node; the data acquisition service can be the Exporter service. For example, the ClickHouse node installs the ClickHouse service and the Exporter service, the gateway node installs the Gateway service and the Exporter service, and the installation / running node installs the ZooKeeper service and the Exporter service. Each node also has diagnostic command scripts configured. For example, the ClickHouse node has the ckService.sh diagnostic command script, the gateway node has the gwService.sh diagnostic command script, and the installation / running node has the zkService.sh diagnostic command script. By running these diagnostic command scripts, the health status of the services can be diagnosed in real time.
[0127] The indicator collection component Prometheus can collect service health information every 15 or 30 seconds. This application does not limit the collection frequency of the indicator collection component.
[0128] For example, if the self-healing window lasts for 5 minutes (this application does not limit this), and the indicator collection component collects the health information of the service every 30 seconds, then the self-healing window may include 10 collection moments.
[0129] All nodes in the target cluster can simultaneously determine whether they have a fault and the type of fault. All nodes in all clusters managed by the management system can also simultaneously determine whether they have a fault and the type of fault.
[0130] For any given time point, the health information can indicate whether a specific service is healthy.
[0131] Setting a self-healing window can improve the accuracy of data collection, thereby improving the accuracy of fault diagnosis.
[0132] S402. Based on the health information of a specific service within the self-healing window and / or the status information of the data collection service, execute a diagnostic command. The diagnostic command is used to diagnose the health status of a specific service in real time.
[0133] In one possible implementation, a diagnostic command is executed based on the health information of a specific service within the self-healing window and / or the status information of the data collection service, including any one of the following:
[0134] ① If the health information of a specific service in the self-healing window is incomplete, and the status information of the data collection service indicates that the data collection service is unhealthy, then restart the data collection service and execute the diagnostic command.
[0135] Incomplete health information for a specific service within the self-healing window can mean that there is no corresponding health information at some collection points within the self-healing window.
[0136] If the data collection service is unhealthy, the self-repair function of the data collection service can ensure the integrity of the health information collected and avoid misjudgments caused by collection failure.
[0137] ② If the health information of a specific service in the self-healing window is incomplete, and the status information of the data collection service indicates that the data collection service is healthy, then execute the diagnostic command.
[0138] ③ If the health information of a specific service is complete within the self-healing window, and the health information corresponding to at least one of the multiple collection times indicates that the specific service is unhealthy, then execute the diagnostic command.
[0139] If the health information of a specific service is complete within the self-healing window, and the health information corresponding to multiple collection times all indicate that the specific service is healthy, then the process ends and no diagnostic command needs to be executed.
[0140] S403. If the diagnostic command cannot be executed, the fault type of the node is determined to be a host fault.
[0141] S404. If the diagnostic command is executed successfully and the execution result indicates that the health status of a specific service is unhealthy, then the fault type of the node is determined to be a service fault.
[0142] If the diagnostic command is executed successfully and the result indicates that the health status of the specific service is healthy, the process ends.
[0143] For S403 or S404, after determining the fault type of the node, the fault type information is recorded, which indicates the fault type of the node.
[0144] exist Figure 4 In the illustrated embodiment, a multi-determination mechanism is used to improve the accuracy of node fault type determination.
[0145] Based on the above, in one possible implementation, the self-healing parameters corresponding to the target cluster can be determined according to the services applicable to the target cluster. The self-healing parameters include at least one of the following: the duration of the self-healing window, the duration of the fault recovery window, the threshold of self-healable fault nodes, the percentage threshold of self-healable fault nodes in the total number of nodes in the cluster, and the repeated execution threshold.
[0146] By flexibly configuring self-healing parameters, the false positive rate can be reduced and the matching degree between the self-healing strategy and the actual scenario can be improved.
[0147] In addition to the aforementioned beneficial effects, the embodiments of this application can also reduce operation and maintenance costs, greatly improve the availability of the management system, reduce human error, enhance the reliability of the management system, and improve operation and maintenance efficiency.
[0148] Figure 5 This is a schematic diagram of a cluster fault handling device provided in an embodiment of this application. Figure 5 As shown, the cluster fault handling device 500 includes: an acquisition module 501, a matching module 502, and a fault recovery module 503. Among them,
[0149] The acquisition module 501 is used to acquire the fault type information and cluster status information of the target node in the target cluster. The fault type information of the target node indicates whether the fault type of the target node is a service fault or a host fault. The cluster status information includes the number of currently faulty nodes and similar fault information. The similar fault information indicates whether the time of the fault occurrence of the target node is within the first fault recovery window. The first fault recovery window is the fault recovery window corresponding to the first fault that has self-healed. The fault of the target node is the same as or similar to the first fault.
[0150] The matching module 502 is used to perform matching operations, which include: matching a first self-healing strategy for the target node's fault based on the target node's fault type information and cluster status information;
[0151] The fault recovery module 503 is used to execute the first self-healing strategy to recover the fault of the target node.
[0152] In one possible implementation, the matching module 502 is specifically configured to perform any of the following:
[0153] If the fault type information of the target node indicates that the fault type of the target node is a service fault, and the number of currently faulty nodes is less than or equal to the threshold of self-healing faulty nodes, and the similar fault information indicates that the time of the fault occurrence of the target node is outside the first fault recovery window, then restarting the service will be matched as the first self-healing strategy.
[0154] If the fault type information of the target node indicates that the fault type of the target node is a host fault, and the number of currently faulty nodes is less than or equal to the threshold of self-healing faulty nodes, and the similar fault information indicates that the time of the fault occurrence of the target node is outside the first fault recovery window, then restarting the operating system will be matched as the first self-healing strategy.
[0155] In one possible implementation, the matching module 502 is specifically used for:
[0156] If the number of current faulty nodes is greater than the threshold of self-healing faulty nodes, or if similar fault information indicates that the fault of the target node occurred within the first fault recovery window, then manual operation will be triggered to match it with the first self-healing strategy.
[0157] In one possible implementation, the device 500 further includes a switching module 504 and a repetition module 505, wherein,
[0158] The switching module 504 is used to perform a switching operation if the first self-healing strategy fails. The switching operation includes: switching the fault self-healing strategy of the target node to the second self-healing strategy according to the preset strategy upgrade path. The level of the second self-healing strategy is higher than that of the first self-healing strategy.
[0159] The repeat execution module 505 is used to repeatedly execute matching and switching operations until the target node's fault is recovered or manual operation is triggered.
[0160] In one possible implementation, the switching module 504 is specifically used for:
[0161] If the first self-healing strategy fails to execute on the first attempt, the first self-healing strategy will be executed repeatedly until the number of times the first self-healing strategy is executed reaches the repeated execution threshold and the first self-healing strategy fails to execute, at which point a switching operation will be performed.
[0162] In one possible implementation, the target cluster includes multiple nodes, each node deploying a data acquisition service and a specific service; the device 500 also includes a diagnostic module 506 and a determination module 507, wherein...
[0163] The acquisition module 501 is also used to acquire, for any node in the target cluster, the health information of a specific service in the node within the self-healing window and the status information of the data acquisition service. The self-healing window includes multiple acquisition times, the health information of the specific service in the self-healing window includes the health information corresponding to multiple acquisition times, and the status information of the data acquisition service indicates whether the data acquisition service is healthy.
[0164] The diagnostic module 506 is used to execute diagnostic commands based on the health information of a specific service within the self-healing window and / or the status information of the data collection service. The diagnostic commands are used to diagnose the health status of a specific service in real time.
[0165] The determination module 507 is used to determine the fault type of the node as a host fault if the diagnostic command cannot be executed; and to determine the fault type of the node as a service fault if the diagnostic command is executed successfully and the execution result indicates that the health status of a specific service is unhealthy.
[0166] In one possible implementation, the diagnostic module 506 is specifically configured to perform any of the following:
[0167] If the health information of a specific service is incomplete within the self-healing window, and the status information of the data collection service indicates that the data collection service is unhealthy, then restart the data collection service and execute the diagnostic command.
[0168] If the health information of a specific service within the self-healing window is incomplete, and the status information of the data collection service indicates that the data collection service is healthy, then execute the diagnostic command.
[0169] If the health information of a specific service is complete within the self-healing window, and the health information corresponding to at least one of the multiple data collection times indicates that the specific service is unhealthy, then a diagnostic command is executed.
[0170] In one possible implementation, the determining module 507 is further configured to:
[0171] Based on the business applicable to the target cluster, determine the self-healing parameters corresponding to the target cluster. The self-healing parameters include at least one of the following: the duration of the self-healing window, the duration of the fault recovery window, the threshold of self-healable fault nodes, the percentage threshold of self-healable fault nodes in the total number of nodes in the cluster, and the repeated execution threshold.
[0172] The cluster fault handling device provided in this application embodiment can be used to execute the technical solution of the cluster fault handling method in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.
[0173] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the fault recovery module 503 can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and its functions can be called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through the integrated logic circuits in the hardware of the processor element or through software instructions.
[0174] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device may include a transceiver 601, a processor 602, and a memory 603.
[0175] Processor 602 executes computer execution instructions stored in memory, causing processor 602 to perform the scheme in the above embodiments. Processor 602 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0176] The memory 603 is connected to the processor 602 via the system bus and completes communication between them. The memory 703 is used to store computer program instructions.
[0177] Transceiver 601 can be used to obtain fault type information and cluster status information of target nodes in the target cluster.
[0178] The system bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus. The system bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the figures, but this does not indicate that there is only one bus or a single type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write databases, and read-only databases).
[0179] The memory may include random access memory (RAM) and may also include non-volatile memory (VNM).
[0180] This application also provides a chip for executing instructions, which is used to execute the cluster fault handling method described in the above embodiments.
[0181] This application also provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the cluster fault handling method described in the above embodiments.
[0182] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the cluster fault handling method in the above embodiments.
[0183] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.
[0184] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.
[0185] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.
[0186] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.
[0187] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.
[0188] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.
[0189] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A cluster fault handling method, characterized in that, include: Obtain the fault type information and cluster status information of the target node in the target cluster. The fault type information of the target node indicates whether the fault type of the target node is a service fault or a host fault. The cluster status information includes the number of currently faulty nodes and similar fault information. The similar fault information indicates whether the time of the fault occurrence of the target node is within the first fault recovery window. The first fault recovery window is the fault recovery window corresponding to the first fault that has self-healed. The fault of the target node is the same as or similar to the first fault. Perform a matching operation, the matching operation including: matching a first self-healing strategy for the fault of the target node based on the fault type information of the target node and the cluster status information; The first self-healing strategy is executed to recover the target node from failure.
2. The method according to claim 1, characterized in that, The matching operation includes any one of the following: If the fault type information of the target node indicates that the fault type of the target node is the service fault, and the number of the current fault nodes is less than or equal to the threshold of self-healing fault nodes, and the similar fault information indicates that the fault occurrence time of the target node is outside the first fault recovery window, then restarting the service will be matched as the first self-healing strategy. If the fault type information of the target node indicates that the fault type of the target node is the host fault, and the number of currently faulty nodes is less than or equal to the threshold of self-healing faulty nodes, and the similar fault information indicates that the time of the fault occurrence of the target node is outside the first fault recovery window, then restarting the operating system will be matched as the first self-healing strategy.
3. The method according to claim 1, characterized in that, The matching operation includes: If the number of currently faulty nodes is greater than the threshold of self-healing faulty nodes, or if the similar fault information indicates that the fault occurrence time of the target node is within the first fault recovery window, then manual operation will be triggered to match the first self-healing strategy.
4. The method according to claim 1 or 2, characterized in that, The method further includes: If the first self-healing strategy fails, a switching operation is performed. The switching operation includes: switching the fault self-healing strategy of the target node to a second self-healing strategy according to a preset strategy upgrade path. The level of the second self-healing strategy is higher than that of the first self-healing strategy. Repeat the matching and switching operations until the fault of the target node is recovered or manual operation is triggered.
5. The method according to claim 4, characterized in that, If the first self-healing strategy fails, a switching operation is performed, including: If the first self-healing strategy fails to execute for the first time, the first self-healing strategy is executed repeatedly until the number of times the first self-healing strategy is executed reaches the repeated execution threshold and the first self-healing strategy fails to execute, then the switching operation is executed.
6. The method according to any one of claims 1-5, characterized in that, The target cluster comprises multiple nodes, each node deploying a data acquisition service and a specific service; the method further includes: For any node in the target cluster, obtain the health information of a specific service in the node within a self-healing window and the status information of the data collection service. The self-healing window includes multiple collection times, the health information of the specific service within the self-healing window includes the health information corresponding to the multiple collection times, and the status information of the data collection service indicates whether the data collection service is healthy. Based on the health information of the specific service within the self-healing window and / or the status information of the data acquisition service, a diagnostic command is executed, which is used to diagnose the health status of the specific service in real time. If the diagnostic command cannot be executed, the fault type of the node is determined to be a host fault; If the diagnostic command is successfully executed and the execution result indicates that the health status of the specific service is unhealthy, then the fault type of the node is determined to be a service fault.
7. The method according to claim 6, characterized in that, The step of executing a diagnostic command based on the health information of the specific service within the self-healing window and / or the status information of the data collection service includes any one of the following: If the health information of the specific service within the self-healing window is incomplete, and the status information of the data collection service indicates that the data collection service is unhealthy, then the data collection service is restarted and the diagnostic command is executed. If the health information of the specific service within the self-healing window is incomplete, and the status information of the data collection service indicates that the data collection service is healthy, then the diagnostic command is executed. If the health information of the specific service is complete within the self-healing window, and the health information corresponding to at least one of the multiple collection times indicates that the specific service is unhealthy, then the diagnostic command is executed.
8. The method according to any one of claims 1-7, characterized in that, The method further includes: Based on the services applicable to the target cluster, determine the self-healing parameters corresponding to the target cluster. The self-healing parameters include at least one of the following: the duration of the self-healing window, the duration of the fault recovery window, the threshold of self-healable fault nodes, the percentage threshold of self-healable fault nodes in the total number of nodes in the cluster, and the repeated execution threshold.
9. A cluster fault handling device, characterized in that, include: The acquisition module is used to acquire fault type information and cluster status information of target nodes in the target cluster. The fault type information of the target node indicates that the fault type of the target node is a service fault or a host fault. The cluster status information includes the number of currently faulty nodes and similar fault information. The similar fault information indicates whether the time of occurrence of the fault of the target node is within the first fault recovery window. The first fault recovery window is the fault recovery window corresponding to the first fault that has self-healed. The fault of the target node is the same as or similar to the first fault. A matching module is used to perform a matching operation, the matching operation including: matching a first self-healing strategy for the fault of the target node based on the fault type information of the target node and the cluster status information; The fault recovery module is used to execute the first self-healing strategy to recover the fault of the target node.
10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-8.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.
12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-8.