A cluster node fault detection method and electronic device

Abstract

The present invention discloses a cluster node fault detection method and electronic device. The method comprises: obtaining real-time parameters of cluster nodes under multiple preset indicators; determining whether the real-time parameters meet corresponding judgment conditions; determining whether a first preset condition is met, the first preset condition being that the number of target indicators among multiple preset indicators is greater than a preset number threshold, and the real-time parameters of the cluster nodes under the target indicators do not meet the corresponding judgment conditions; obtaining historical data of the cluster nodes under the target indicators; determining whether a second preset condition is met, the second preset condition being that the multiple between the real-time parameters of the cluster nodes and the historical data exceeds a preset multiple threshold; when at least one of the first preset condition and the second preset condition is met, determining that the cluster node has a fault. The present invention improves the accuracy of fault judgment by comprehensively considering the real-time parameters of the cluster nodes under multiple preset indicators, the corresponding preset conditions, and the corresponding historical data.

Description

Technical Field

[0001] The present invention belongs to the technical field of clusters, and in particular relates to a cluster node fault detection method and electronic equipment. Background Art

[0002] In the actual product development process, in the large-scale testing of cluster stability, automated testing and anti-degradation control for cluster products is crucial. We need a method to quickly locate and determine the cause of performance degradation and quickly iterate and update, continuously improve product quality and testing efficiency, and ensure continuous stability testing.

[0003] First, for large-scale clusters, the anti-degradation monitoring needs to cover the situation of each node. When an individual node is abnormal, it is necessary to discover and shut down the current node in advance to avoid affecting the overall service of the cluster. This places high demands on early discovery. In some scenarios, pinging does not mean that the service is available. It is necessary to fully cover and improve anti-degradation monitoring, recovery, and iterative improvement from the application layer to the network layer.

[0004] In the existing cluster node detection scheme, various parameters of the cluster node are usually detected, and the corresponding threshold is used to determine whether the cluster node is faulty. However, the single threshold method is prone to misjudgment, which reduces the accuracy of the detection result. Summary of the invention

[0005] An object of the present invention is to provide a cluster node fault detection method and electronic device, which can solve the technical problem of low accuracy of detection results in the prior art.

[0006] According to a first aspect of the present invention, a cluster node fault detection method is provided, the method comprising:

[0007] Get the real-time parameters of cluster nodes under various preset indicators;

[0008] Determine whether the real-time parameters of the cluster node under each of the preset indicators meet the determination conditions corresponding to the preset indicators;

[0009] Determine whether a first preset condition is met, wherein the first preset condition is that the number of target indicators among the multiple preset indicators is greater than a preset number threshold, and the real-time parameter of the cluster node under the target indicator does not meet the determination condition corresponding to the target indicator;

[0010] Obtain historical data of cluster nodes under the target indicator;

[0011] Determine whether a second preset condition is met, wherein the second preset condition is that a multiple of the real-time parameter of the cluster node under the target indicator and the historical data of the cluster node under the target indicator exceeds a preset multiple threshold;

[0012] When at least one of the first preset condition and the second preset condition is met, it is determined that the cluster node fails.

[0013] Optionally, the preset indicators include network-related indicators, and the real-time parameters of the cluster nodes under the preset indicators include packet loss rate and port reachability;

[0014] The determining whether the real-time parameter of the cluster node under each of the preset indicators satisfies the determination condition corresponding to the preset indicator includes:

[0015] When the packet loss rate is greater than a preset packet loss rate and / or the port reachability is unreachable, it is determined that the real-time parameter of the cluster node under the network-related indicator does not satisfy a determination condition corresponding to the network-related indicator.

[0016] Optionally, the preset indicator includes a business-related indicator, and the real-time parameters of the cluster node under the preset indicator include a business request success rate and a business request response time;

[0017] The determining whether the real-time parameter of the cluster node under each of the preset indicators satisfies the determination condition corresponding to the preset indicator includes:

[0018] When the service request success rate is lower than a preset service request success rate threshold and / or the service request response time exceeds a preset response time, it is determined that the real-time parameter of the cluster node under the service-related indicator does not meet the determination condition corresponding to the service-related indicator.

[0019] Optionally, the preset indicator includes a resource utilization indicator, the real-time parameter of the cluster node under the preset indicator includes the real-time resource utilization of the cluster node, the historical data of the cluster node under the target indicator includes the historical resource utilization in the same time period, and the resource utilization includes CPU utilization, memory utilization, and disk utilization;

[0020] The determining whether the second preset condition is met includes:

[0021] Calculate the first multiple of the real-time resource utilization of the cluster node exceeding the historical resource utilization in the same time period;

[0022] If the first multiple exceeds the preset multiple threshold corresponding to the resource utilization index, it is determined that the second preset condition is met.

[0023] Optionally, the preset indicator includes a process indicator, and the real-time parameter of the cluster node under the preset indicator includes the number of changes of a key process ID;

[0024] The determining whether the real-time parameter of the cluster node under each of the preset indicators satisfies the determination condition corresponding to the preset indicator includes:

[0025] When the number of changes of the key process ID is greater than or equal to 1, it is determined that the real-time parameter of the cluster node under the process indicator does not satisfy the determination condition corresponding to the process indicator.

[0026] Optionally, the preset indicator includes a tracking point business indicator, and the real-time parameters of the cluster node under the preset indicator include the number of times the tracking point business returns data within a preset time period and the increment of the number of times the tracking point business returns data within the preset time period, wherein the increment of the number of times the tracking point business returns data within the preset time period is the difference between the number of times the tracking point business returns data within the preset time period and the number of times the tracking point business returns data in the previous time period;

[0027] The determining whether the real-time parameter of the cluster node under each of the preset indicators satisfies the determination condition corresponding to the preset indicator includes:

[0028] If the number of data returned by the tracking service within the preset time period is 0, it is determined that the real-time parameter of the cluster node under the tracking service indicator does not meet the determination condition corresponding to the tracking service indicator;

[0029] If the increment of the number of times the tracking point service returns data within the preset time period is less than the preset increment threshold, it is determined that the real-time parameters of the cluster node under the tracking point service indicator do not meet the judgment conditions corresponding to the tracking point service indicator.

[0030] Optionally, the preset indicator includes an error log indicator of a key process, and the real-time parameter of the cluster node under the error log indicator includes the number of newly added error logs of the key process;

[0031] The determining whether the real-time parameter of the cluster node under each of the preset indicators satisfies the determination condition corresponding to the preset indicator includes:

[0032] When the number of newly added error logs of the key process is greater than 0, it is determined that the real-time parameter of the cluster node under the error log indicator of the key process does not satisfy the determination condition corresponding to the error log indicator of the key process.

[0033] Optionally, the preset indicator includes an error log indicator of a cluster node, the real-time parameter of the cluster node under the error log indicator of the cluster node includes the number of new error logs of the cluster node, and the historical data of the cluster node under the preset indicator includes the number of new error logs of the cluster node in the same time period;

[0034] The determining whether the second preset condition is met includes:

[0035] Calculate the second multiple of the number of new error logs of the cluster node exceeding the number of new error logs of the cluster node in the same time period;

[0036] If the second multiple exceeds the multiple threshold corresponding to the error log indicator of the cluster node, it is determined that the second preset condition is met.

[0037] Optionally, the method further comprises:

[0038] In the case where it is determined that the cluster node fails, determining whether there is a pre-configured solution corresponding to the failure type of the cluster node;

[0039] If so, execute a pre-configured repair plan corresponding to the fault type of the cluster node and output an alarm log;

[0040] If it does not exist, execute the pre-configured general repair plan and output an alarm notification based on the repair result.

[0041] According to a second aspect of the present invention, there is provided an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions executable by the processor, and when the programs or instructions are executed by the processor, the cluster node fault detection method as described in any one of the first aspect of the present invention is implemented.

[0042] The beneficial effects of the present invention are as follows: the present invention comprehensively considers the real-time parameters of the cluster nodes under a variety of preset indicators, the corresponding preset conditions and the corresponding historical data, determines whether a cluster node has a fault according to the number of target indicators that do not meet the preset conditions and the multiples by which the real-time parameters exceed the corresponding historical data, and improves the accuracy of fault judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of a cluster node fault detection method in one embodiment of the present invention.

[0044] Figure 2 It is a schematic diagram of testing the packet loss rate in one embodiment of the present invention.

[0045] Figure 3It is a schematic diagram of testing the packet loss rate in another embodiment of the present invention.

[0046] Figure 4 It is a schematic diagram of testing port reachability in one embodiment of the present invention.

[0047] Figure 5 It is a schematic diagram of testing port reachability in another embodiment of the present invention.

[0048] Figure 6 It is a schematic diagram of testing the success rate of a service request in one embodiment of the present invention.

[0049] Figure 7 It is a schematic diagram of a cluster node fault detection device in one embodiment of the present invention.

[0050] Figure 8 It is a flow chart of cluster node failure detection in one embodiment of the present invention. DETAILED DESCRIPTION

[0051] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0052] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0053] The techniques, methods and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods and devices should be considered as part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0054] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0055] In the specification and claims of the present invention, if the term "first" or "second" is involved, it may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "plurality" means two or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally means that the related objects are in an "or" relationship.

[0056] like Figure 1As shown, this embodiment introduces a cluster node fault detection method, which includes steps 1100-1600.

[0057] Step 1100: Obtain real-time parameters of cluster nodes under multiple preset indicators.

[0058] Preset indicators can include network-related indicators, business-related indicators, resource utilization indicators, process indicators, embedded business indicators, error log indicators of key processes, and error log indicators of cluster nodes. The cluster can be a WAF (Web Application Firewall) cluster or other clusters.

[0059] Step 1200: Determine whether the real-time parameters of the cluster node under each of the preset indicators meet the determination conditions corresponding to the preset indicators.

[0060] Pre-configure corresponding judgment conditions for each preset indicator. For example, network-related indicators may include network connectivity and port reachability, and the corresponding judgment conditions are network connectivity and port reachability. For another example, the judgment condition corresponding to the resource utilization indicator may be whether the resource utilization exceeds the corresponding threshold.

[0061] Step 1300: Determine whether a first preset condition is met, wherein the first preset condition is that the number of target indicators among the multiple preset indicators is greater than a preset number threshold, and the real-time parameters of the cluster node under the target indicator do not meet the judgment condition corresponding to the target indicator.

[0062] Determine whether the real-time parameters of the cluster node under each preset indicator meet the judgment condition corresponding to the preset indicator. If not, then the preset indicator is the target indicator. Count the number of target indicators among all preset indicators. If the number of target indicators is greater than the preset number threshold, then it is considered that the first preset condition is met. The preset number threshold is usually set to 1, which means that if the real-time parameters of the cluster node under two or more preset indicators do not meet the corresponding judgment condition, then it is considered that the first preset condition is met.

[0063] For example, the preset indicators include network-related indicators, business-related indicators, resource utilization indicators, process indicators, tracking business indicators, error log indicators of key processes, and error log indicators of cluster nodes. The preset indicators include a total of 7 indicators. If the real-time parameters of the cluster node under the network-related indicators, business-related indicators, and resource utilization indicators do not meet the corresponding judgment conditions, and the real-time parameters of the cluster node under other indicators meet the corresponding judgment conditions, then the network-related indicators and business-related indicators are target indicators, and the number of target indicators is 3. The preset quantity threshold is 1. At this time, the number of target indicators is greater than the preset quantity threshold, then it is considered that the first preset condition is met.

[0064] Step 1400: Obtain historical data of the cluster node under the target indicator.

[0065] Historical data can be data from the same time period as the real-time parameters. For example, if the real-time parameters of the cluster node under the preset indicator are data from 8:00-10:00 am, then the historical data can be data from 8:00-10:00 am a few days ago.

[0066] Step 1500: Determine whether a second preset condition is met, wherein the second preset condition is that a multiple of a real-time parameter of the cluster node under the target indicator and historical data of the cluster node under the target indicator exceeds a preset multiple threshold.

[0067] Calculate the multiple between the real-time parameter and the corresponding historical data. For example, if the preset indicator is resource utilization, the real-time parameter is 60%, and the corresponding historical data is 20%, then the multiple is 3. The preset multiple threshold is 2, and if it exceeds the preset multiple threshold, it is determined that the second preset condition is met.

[0068] Step 1600: When at least one of the first preset condition and the second preset condition is met, determine that the cluster node fails.

[0069] If at least one of the first preset condition and the second preset condition is met, it is considered that the cluster node fails.

[0070] The present invention comprehensively considers the real-time parameters of the cluster nodes under multiple preset indicators, the corresponding preset conditions and the corresponding historical data, determines whether the cluster node has a fault according to the number of target indicators that do not meet the preset conditions and the multiples by which the real-time parameters exceed the corresponding historical data, and improves the accuracy of fault judgment.

[0071] In this embodiment, the preset indicators include network-related indicators, and the real-time parameters of the cluster nodes under the preset indicators include packet loss rate and port reachability.

[0072] The step 1200 includes: when the packet loss rate is greater than a preset packet loss rate and / or the port reachability is unreachable, determining that the real-time parameter of the cluster node under the network-related indicator does not satisfy a determination condition corresponding to the network-related indicator.

[0073] You can obtain the corresponding real-time parameters by sending access test requests to cluster nodes. For example, you can use the ping command to send a network access request to a cluster node to test the network connectivity of the cluster node. Figure 2 As shown in the figure, 7 data packets were sent to the cluster nodes, and all 7 data packets were received by the cluster nodes, so the packet loss rate is 0%. Figure 3 As shown in the figure, 92 data packets were sent to the cluster nodes, but none of the cluster nodes received them, and the packet loss rate was 100%.

[0074] You can use the telnet command to test the port reachability. Determine the port reachability based on the test results. Figure 4 As shown in the figure, connected means the connection is successful and the port is open for service. Figure 5 As shown, refused means the connection failed and the port is not open.

[0075] If the current packet loss rate of the cluster node is greater than the preset packet loss rate, it is considered that the judgment condition is not met. For example, if the current packet loss rate of the cluster node is 10% and the preset packet loss rate is 5%, then the current packet loss rate of the cluster node is greater than the preset packet loss rate, and it is determined that the real-time parameters of the cluster node under the network-related indicators do not meet the judgment conditions corresponding to the network-related indicators. If the test shows that the current port of the cluster node is unreachable, it is also determined that the real-time parameters of the cluster node under the network-related indicators do not meet the judgment conditions corresponding to the network-related indicators.

[0076] In this embodiment, the preset indicators include service-related indicators, and the real-time parameters of the cluster nodes under the preset indicators include service request success rate and service request response time.

[0077] The step 1200 includes: when the business request success rate is lower than a preset business request success rate threshold and / or the business request response time exceeds a preset response time, determining that the real-time parameter of the cluster node under the business-related indicator does not meet the judgment condition corresponding to the business-related indicator.

[0078] You can initiate a service request to the cluster node, and the cluster node will respond after receiving the service request, and obtain the service request success rate and service request response time. Figure 6 As shown in the figure, after initiating the service request, the feedback 200ok is obtained, which means that the request is successful.

[0079] For example, through testing, it is found that the business request success rate is 90%, and the preset business request success rate threshold is 95%. Then, the business request success rate is lower than the preset business request success rate threshold, and it is considered that the real-time parameters of the cluster node under the business-related indicators do not meet the judgment conditions corresponding to the business-related indicators.

[0080] For another example, through testing, it is found that the business request response time is 3000ms, and the preset response time is 1000ms. The business request response time exceeds the preset response time. It is also considered that the real-time parameters of the cluster node under the business-related indicators do not meet the judgment conditions corresponding to the business-related indicators.

[0081] In this embodiment, the preset indicators include resource utilization indicators, the real-time parameters of the cluster nodes under the preset indicators include the real-time resource utilization of the cluster nodes, the historical data of the cluster nodes under the target indicators include the historical resource utilization in the same time period, and the resource utilization includes CPU utilization, memory utilization, and disk utilization.

[0082] The step 1500 includes: calculating a first multiple of the real-time resource utilization of the cluster node exceeding the historical resource utilization of the same time period; if the first multiple exceeds a preset multiple threshold corresponding to the resource utilization indicator, determining that the second preset condition is satisfied.

[0083] The resource utilization of cluster nodes fluctuates greatly in different time periods during the day. For example, starting from 8:00 in the morning, the number of people going to work increases, and the CPU (Central Processing Unit) utilization may reach 50%, and continue until 17:00 in the afternoon when get off work is over. After get off work, the CPU utilization drops sharply and may be only 20%. Similarly, memory utilization and disk utilization will also fluctuate between different time periods.

[0084] After obtaining the current resource utilization of the cluster node, calculate the first multiple of the historical resource utilization in the same time period. For example, if the current resource utilization of the cluster node is 80% and the historical resource utilization in the same time period is 20%, then the current resource utilization of the cluster node exceeds the historical resource utilization in the same time period by 60%, and the multiple is 3 times. If the preset multiple threshold is 1, it is determined that the second preset condition is met.

[0085] If only a single threshold is used and an alarm is sounded when the resource utilization exceeds the threshold, false alarms may occur. The present invention calculates the first multiple of the real-time resource utilization of the cluster node exceeding the historical resource utilization in the same time period, determines whether the second preset condition is met according to the first multiple, and then determines whether the cluster node has a fault, thereby improving the accuracy of fault judgment.

[0086] In this embodiment, the preset indicator includes a process indicator, and the real-time parameter of the cluster node under the preset indicator includes the number of changes of the key process ID.

[0087] The step 1200 includes: when the number of changes of the key process ID is greater than or equal to 1, determining that the real-time parameter of the cluster node under the process indicator does not satisfy the determination condition corresponding to the process indicator.

[0088] A critical process is a process that needs to run continuously. The critical process can be set according to the actual business. For example, a security-related process can be set as a critical process. If the ID (identification) of a critical process changes, it means that the critical process has been restarted, which is not as expected. The number of changes in the critical process ID indicates the number of restarts.

[0089] If the number of changes in the key process ID is greater than or equal to 1, it means that the key process has been restarted. In this case, it is considered that the real-time parameters of the cluster node under the process indicator do not meet the judgment conditions corresponding to the process indicator.

[0090] In this embodiment, the preset indicators include tracking point business indicators, and the real-time parameters of the cluster nodes under the preset indicators include the number of times the tracking point business returns data within a preset time period and the increment of the number of times the tracking point business returns data within the preset time period, wherein the increment of the number of times the tracking point business returns data within the preset time period is the difference between the number of times the tracking point business returns data within the preset time period and the number of times the tracking point business returns data in the previous time period.

[0091] The step 1200 includes: if the number of data returned by the tracking business within the preset time period is 0, determining that the real-time parameters of the cluster node under the tracking business indicator do not meet the judgment conditions corresponding to the tracking business indicator; if the increment of the number of data returned by the tracking business within the preset time period is less than the preset increment threshold, determining that the real-time parameters of the cluster node under the tracking business indicator do not meet the judgment conditions corresponding to the tracking business indicator.

[0092] The tracking service is triggered within the preset time period. According to the pre-configured rules, the tracking service returns a data each time it is triggered. If the number of data returned by the tracking service within the preset time period is 0, it means that an exception has occurred, and it is determined that the real-time parameters of the cluster node under the tracking service indicator do not meet the judgment conditions corresponding to the tracking service indicator.

[0093] The increment of the number of times the tracking service returns data represents the difference between the number of times the tracking service returns data in two adjacent time periods. For example, if the number of times the tracking service returns data on the same day is 300, and the number of times the tracking service returns data on the previous day is 100, then the increment of the number of times the tracking service returns data is 200. If the preset increment threshold is 50, then it is determined that the corresponding judgment condition is not met.

[0094] In this embodiment, the preset indicator includes an error log indicator of a key process, and the real-time parameter of the cluster node under the error log indicator includes the number of newly added error logs of the key process.

[0095] The step 1200 includes: when the number of newly added error logs of the key process is greater than 0, determining that the real-time parameter of the cluster node under the error log indicator of the key process does not meet the determination condition corresponding to the error log indicator of the key process.

[0096] Error logs include error logs of key processes and other logs. If an abnormal situation occurs during cluster operation, a corresponding error log will be given. Not all situations where error logs appear will affect the function of the cluster. For example, for some modules with lower importance, when these modules have abnormalities, the normal function of the cluster will not be affected.

[0097] For critical processes, if corresponding error logs appear, it will inevitably affect the normal function of the cluster. When the number of new error logs of critical processes is greater than 0, it means that the critical process has an abnormality. In this case, it is determined that the real-time parameters of the cluster node under the error log indicator of the critical process do not meet the judgment conditions corresponding to the error log indicator of the critical process.

[0098] In this embodiment, the preset indicator includes the error log indicator of the cluster node, the real-time parameters of the cluster node under the error log indicator of the cluster node include the new number of error logs of the cluster node, and the historical data of the cluster node under the preset indicator includes the new number of error logs of the cluster node in the same time period.

[0099] The step 1500 includes: calculating the second multiple of the number of new error logs of the cluster node exceeding the number of new error logs of the cluster node in the same time period; if the second multiple exceeds the multiple threshold corresponding to the error log indicator of the cluster node, determining that the second preset condition is met.

[0100] After obtaining the number of new error logs of cluster nodes, calculate the second multiple of the number of new error logs of cluster nodes exceeding the number of new error logs of cluster nodes in the same time period. For example, if the number of new error logs of cluster nodes on the day is 100, and the number of new error logs per day before is 10, then the number of error logs on the day exceeds the number of error logs per day before by 90, and the second multiple is 9 times. The multiple threshold corresponding to the error log indicator is 2, then it is determined that the second preset condition is met.

[0101] In this embodiment, the method further includes steps 1700-1900.

[0102] Step 1700: When it is determined that the cluster node fails, determine whether there is a pre-configured solution corresponding to the failure type of the cluster node.

[0103] Step 1800: If so, execute a pre-configured repair plan corresponding to the fault type of the cluster node and output an alarm log.

[0104] Step 1900: If it does not exist, execute the pre-configured general repair solution and output an alarm notification based on the repair result.

[0105] Usually, the staff will pre-determine the possible fault types of the cluster and configure corresponding solutions for each fault type. When a cluster node fault is detected, determine whether there is a corresponding solution. If so, execute the corresponding solution to repair it. Solutions corresponding to 20 fault types are pre-configured. When the fault type of the cluster node is one of the above 20 fault types, the corresponding solution is executed.

[0106] In order to deal with unexpected failures that may occur, a general repair plan is also pre-configured. When an unknown type of failure occurs in a cluster node, the general repair plan is executed. The repair results are sent to the staff through an alarm notification for subsequent manual processing.

[0107] This patent pre-configures solutions corresponding to the fault types and executes corresponding solutions to repair cluster nodes when faults occur. It can handle faults in a timely manner and minimize the impact of the faults.

[0108] like Figure 7 As shown, this embodiment introduces a cluster node fault detection device, which includes a test operation module and an anti-degradation system. The anti-degradation system includes a monitoring module, a degradation analysis module, a self-recovery degradation module, an alarm module, a log module, and a configuration module.

[0109] The monitoring module is used to obtain the real-time parameters of the cluster nodes under a variety of preset indicators, and determine whether the real-time parameters of the cluster nodes under each preset indicator meet the judgment conditions corresponding to the preset indicators.

[0110] The degradation analysis module is used to determine whether a first preset condition and a second preset condition are met, and when at least one of the first preset condition and the second preset condition is met, it is determined that a cluster node fails.

[0111] The degradation analysis module is also used to call the alarm module to output alarm information when it is determined that a cluster node has a fault, and call the self-recovery degradation module to execute the corresponding solution. If there is a solution corresponding to the fault type, the self-recovery degradation module executes the solution. If there is no solution corresponding to the fault type, the self-recovery degradation module executes the general repair solution. After the fault is repaired, the self-recovery degradation module calls the alarm module to issue an alarm notification and calls the log module to record the fault repair process. The configuration module can be used to configure each threshold.

[0112] The test operation module can issue tasks to the anti-degradation system, and the anti-degradation system returns the alarm results to the test operation module.

[0113] like Figure 8 As shown in the figure, the test operation module first issues the monitoring task. Then the monitoring module monitors the cluster nodes to determine whether the threshold is reached, that is, whether a fault occurs. If a fault occurs, the degradation analysis module is entered to determine whether there is a corresponding preset solution. If so, the self-recovery degradation module executes the corresponding solution. If not, the general repair solution is executed.

[0114] This embodiment introduces an electronic device, including a processor and a memory, wherein the memory stores programs or instructions executable by the processor, and when the programs or instructions are executed by the processor, the cluster node fault detection method described in any embodiment of the present invention is implemented.

[0115] Although some specific embodiments of the present invention have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It will be appreciated by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

[0116] Those of ordinary skill in the art will appreciate that the modules and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and equipment can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0118] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0119] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0120] In addition, each functional module in the embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0121] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the energy-saving signal sending / receiving method of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0122] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the present application.

[0123] It should be understood that the size of the sequence number of each step in the content of the invention and the embodiments of the present invention does not absolutely mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention. For the purpose of example and description, the foregoing description of the implementation of the present disclosure has been given. The foregoing description is not exhaustive and is not intended to limit the present disclosure to the exact form disclosed. Various deformations and modifications may exist according to the above teachings, or various deformations and modifications may be obtained from the practice of the present disclosure. These embodiments are selected and described to illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can use the present disclosure in various embodiments and various modifications suitable for the specific purpose conceived.

Claims

1. A cluster node fault detection method, characterized in that: The method comprises: Get the real-time parameters of cluster nodes under various preset indicators; Determine whether the real-time parameters of the cluster node under each of the preset indicators meet the determination conditions corresponding to the preset indicators; Determine whether a first preset condition is met, wherein the first preset condition is that the number of target indicators among the multiple preset indicators is greater than a preset number threshold, and the real-time parameter of the cluster node under the target indicator does not meet the determination condition corresponding to the target indicator; Obtain historical data of cluster nodes under the target indicator; Determine whether a second preset condition is met, wherein the second preset condition is that a multiple of the real-time parameter of the cluster node under the target indicator and the historical data of the cluster node under the target indicator exceeds a preset multiple threshold; When at least one of the first preset condition and the second preset condition is met, it is determined that the cluster node fails.

2. The method according to claim 1, characterized in that: The preset indicators include network-related indicators, and the real-time parameters of the cluster nodes under the preset indicators include packet loss rate and port reachability; The determining whether the real-time parameter of the cluster node under each of the preset indicators satisfies the determination condition corresponding to the preset indicator includes: When the packet loss rate is greater than a preset packet loss rate and / or the port reachability is unreachable, it is determined that the real-time parameter of the cluster node under the network-related indicator does not satisfy a determination condition corresponding to the network-related indicator.

3. The method according to claim 1, characterized in that The preset indicators include business-related indicators, and the real-time parameters of the cluster nodes under the preset indicators include business request success rate and business request response time; The determining whether the real-time parameter of the cluster node under each of the preset indicators satisfies the determination condition corresponding to the preset indicator includes: When the service request success rate is lower than a preset service request success rate threshold and / or the service request response time exceeds a preset response time, it is determined that the real-time parameter of the cluster node under the service-related indicator does not meet the determination condition corresponding to the service-related indicator.

4. The method according to claim 1, characterized in that: The preset indicator includes a resource utilization indicator, the real-time parameter of the cluster node under the preset indicator includes the real-time resource utilization of the cluster node, the historical data of the cluster node under the target indicator includes the historical resource utilization of the same time period, and the resource utilization includes CPU utilization, memory utilization, and disk utilization; The determining whether the second preset condition is met includes: Calculate the first multiple of the real-time resource utilization of the cluster node exceeding the historical resource utilization in the same time period; If the first multiple exceeds the preset multiple threshold corresponding to the resource utilization index, it is determined that the second preset condition is met.

5. The method according to claim 1, characterized in that The preset index includes a process index, and the real-time parameter of the cluster node under the preset index includes the number of changes of the key process ID; The determining whether the real-time parameter of the cluster node under each of the preset indicators satisfies the determination condition corresponding to the preset indicator includes: When the number of changes of the key process ID is greater than or equal to 1, it is determined that the real-time parameter of the cluster node under the process indicator does not satisfy the determination condition corresponding to the process indicator.

6. The method according to claim 1, characterized in that The preset indicators include tracking point business indicators, and the real-time parameters of the cluster nodes under the preset indicators include the number of times the tracking point business returns data within a preset time period and the increment of the number of times the tracking point business returns data within the preset time period, wherein the increment of the number of times the tracking point business returns data within the preset time period is the difference between the number of times the tracking point business returns data within the preset time period and the number of times the tracking point business returns data in the previous time period; The determining whether the real-time parameter of the cluster node under each of the preset indicators satisfies the determination condition corresponding to the preset indicator includes: If the number of data returned by the tracking service within the preset time period is 0, it is determined that the real-time parameter of the cluster node under the tracking service indicator does not meet the determination condition corresponding to the tracking service indicator; If the increment of the number of times the tracking point service returns data within the preset time period is less than the preset increment threshold, it is determined that the real-time parameters of the cluster node under the tracking point service indicator do not meet the judgment conditions corresponding to the tracking point service indicator.

7. The method according to claim 1, characterized in that The preset indicator includes an error log indicator of a key process, and the real-time parameter of the cluster node under the error log indicator includes the number of newly added error logs of the key process; The determining whether the real-time parameter of the cluster node under each of the preset indicators satisfies the determination condition corresponding to the preset indicator includes: When the number of newly added error logs of the key process is greater than 0, it is determined that the real-time parameter of the cluster node under the error log indicator of the key process does not satisfy the determination condition corresponding to the error log indicator of the key process.

8. The method according to claim 7, characterized in that The preset indicator includes an error log indicator of a cluster node, the real-time parameter of the cluster node under the error log indicator of the cluster node includes the number of new error logs of the cluster node, and the historical data of the cluster node under the preset indicator includes the number of new error logs of the cluster node in the same time period; The determining whether the second preset condition is met includes: Calculate the second multiple of the number of new error logs of the cluster node exceeding the number of new error logs of the cluster node in the same time period; If the second multiple exceeds the multiple threshold corresponding to the error log indicator of the cluster node, it is determined that the second preset condition is met.

9. The method according to claim 1, characterized in that: The method further comprises: In the case where it is determined that the cluster node fails, determining whether there is a pre-configured solution corresponding to the failure type of the cluster node; If so, execute a pre-configured repair plan corresponding to the fault type of the cluster node and output an alarm log; If it does not exist, execute the pre-configured general repair plan and output an alarm notification based on the repair result.

10. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a program or instruction executable by the processor, and when the program or instruction is executed by the processor, the cluster node fault detection method according to any one of claims 1 to 9 is implemented.

Images