Fault processing method and device, computer equipment, storage medium and computer program product

Abstract

The invention relates to a fault processing method and device, computer equipment, a storage medium and a computer program product, and the method comprises the steps: determining a communication state between a target node and a fault node under the condition that any other node in a distributed cluster is detected to have a fault, and carrying out fault processing on the target node and the fault node under the condition that the communication state is normal. And determining the working state of the system disk on the fault node, and processing the fault according to the working state. According to the method, when the distributed cluster breaks down, the fault node can be recovered to be normally used as much as possible by analyzing the working state of the fault node and remedying the fault, and compared with a method of directly isolating the fault node, the fault remedying method can save the resource cost of the node. Moreover, for the fault nodes which are restored to be normal after being remedied, manpower resources required by manual troubleshooting can be reduced, and thus the fault processing cost is further reduced.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a fault handling method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] A distributed storage system consists of multiple servers forming a server cluster. Each server is a node and becomes the basis of the distributed storage system. When any node in the server cluster fails, the failed node needs to be processed quickly. If the failed node is not processed in time, it will affect the performance of the entire distributed storage system and even lead to problems such as irreversible data loss. Therefore, it is very necessary to efficiently detect and process node failures.

[0003] However, the method for processing node failure in the related art has the problem of high processing cost. Summary of the invention

[0004] Based on this, it is necessary to provide a fault handling method, apparatus, computer equipment, storage medium and computer program product that can reduce the fault handling cost in response to the above technical problems.

[0005] In a first aspect, the present application provides a fault handling method, which is applied to a target node in a distributed cluster, and the method includes:

[0006] In case of detecting a failure of any other node in the distributed cluster, determining the communication status between the target node and the failed node;

[0007] When the communication status is normal, determine the working status of the system disk on the faulty node and handle the fault according to the working status; the working status is used to characterize the read and write capabilities and health status of the system disk; the processing includes isolation or fault remediation operations.

[0008] The detection method provided in the embodiment of the present application determines the communication status between the target node and the faulty node when any other node in the distributed cluster is detected to have failed, and determines the working status of the system disk on the faulty node when the communication status is normal, and handles the fault according to the working status. Among them, the working status is used to characterize the read and write capabilities and health status of the system disk, and the processing includes isolation or fault remediation operations. In the above method, when a fault occurs in the distributed cluster, the faulty node can be restored to normal use as much as possible by analyzing the working status on the faulty node and remediating the fault. Compared with the method of directly isolating the faulty node, the above fault remediation method can save the resource cost of the node. Moreover, for the faulty node that has been restored to normal after remediation, it can also reduce the human resources required for manual troubleshooting, thereby further reducing the cost of fault handling. In addition, the above method can detect from the outside to the inside in multiple dimensions, first detect the communication status between nodes, then detect the working function of the system disk in the node, and then detect the working status of the system disk, which can comprehensively and meticulously identify faults, not only improve the accuracy of fault detection, but also perform corresponding processing for different faults, thereby significantly improving the accuracy of fault handling.

[0009] In one embodiment, the fault is processed according to the working state, including:

[0010] Determine whether the system disk on the faulty node has read and write abnormalities based on the working status;

[0011] If the system disk has a read / write anomaly, a first remedial operation is performed; the first remedial operation includes restarting the faulty node, and processing the fault according to whether the restarted faulty node has a read / write anomaly;

[0012] If there is no reading or writing abnormality on the system disk, the health status of the system disk on the faulty node is determined according to the working status, and the fault is processed according to the health status.

[0013] In one embodiment, the fault is processed according to the health status, including:

[0014] When the health status indicates that the faulty node is in a sub-healthy state, the fault redundancy of the distributed cluster is obtained, and when it is determined that the fault redundancy is greater than a preset redundancy threshold, the fault is processed according to the sub-healthy type corresponding to the sub-healthy state.

[0015] The method described in the embodiment of the present application divides the fault handling process into a read / write exception handling stage and a health status check stage, which can make fault handling more systematic, quickly locate problems, respond quickly, and reduce the time for fault recovery. Moreover, when judging the health status of the system disk, by analyzing the fault redundancy, it is possible to effectively avoid unnecessary migration or restart operations when resource usage has not reached the limit, thereby better utilizing cluster resources and saving costs.

[0016] In one embodiment, the sub-health type is a read-only fault, and the fault is processed according to the sub-health type corresponding to the health state, including:

[0017] Execute a second remedial operation; the second remedial operation includes restarting the faulty node and determining whether the read-only fault of the faulty node after the restart is eliminated. If not, determine whether the number of restarts exceeds the first preset number of restarts, and isolate the faulty node if the number of restarts exceeds the first preset number of restarts.

[0018] The method described in the embodiment of the present application sets an upper limit on the number of restarts to prevent excessive consumption of system resources due to unlimited restart operations, thereby ensuring the stability of the system during fault handling. Moreover, when restarting is invalid, the faulty node is isolated in time to avoid the spread of the fault, thereby protecting the health of the entire system and the security of the data.

[0019] In one embodiment, the sub-health type is a capacity fault, and the fault is processed according to the sub-health type corresponding to the sub-health state, including:

[0020] Execute a third remedial operation; the third remedial operation includes clearing preset files under the faulty node, or prompting the user to clear the files; determine whether the capacity failure of the system disk of the faulty node after clearing is eliminated, and if not, isolate the faulty node.

[0021] The method described in the embodiment of the present application can effectively release storage space by clearing preset files under the faulty node. At the same time, the user is prompted to perform the clearing operation in a timely manner, which can achieve the effect of human-computer interaction and improve the flexibility of fault handling.

[0022] In one embodiment, the sub-health type is a slow disk fault, and the fault is processed according to the sub-health type corresponding to the sub-health state, including:

[0023] Execute the fourth remedial operation; the fourth remedial operation includes: determining whether the system disk on the faulty node is a disk array; if the system disk is a disk array, determining whether the faulty node needs to be isolated based on the number of abnormal alarms of the faulty node; if the system disk is not a disk array, isolating the faulty node.

[0024] In one embodiment, determining whether to isolate the faulty node according to the number of abnormal alarms of the faulty node includes:

[0025] Determine whether the number of abnormal alarms of the faulty node exceeds a preset number threshold;

[0026] If the number of abnormal alarms of the faulty node does not exceed the preset number threshold, the faulty node is restarted, and it is determined whether the read-only fault of the faulty node after the restart is eliminated. If not eliminated, it is determined whether the number of restarts exceeds a second preset number of restarts, and if the number of restarts exceeds the second preset number of restarts, the faulty node is isolated;

[0027] If the number of abnormal alarms of a faulty node exceeds a preset threshold, the faulty node will be isolated.

[0028] The method described in the embodiment of the present application can perform more targeted management and optimize resource usage by judging whether the system disk on the faulty node is a disk array. In addition, by analyzing the number of abnormal alarms when the system disk is a disk array, data support can be provided for preventive maintenance of the system, reducing the risk of sudden failures and further improving the accuracy of fault handling.

[0029] In one embodiment, the method further comprises:

[0030] Determine whether to allow isolation of the faulty node according to the operation status of the target business module when the faulty node fails; the target business module is the business module associated with the fault;

[0031] In the case where the faulty node is allowed to be isolated, the process returns to the step of processing the fault according to the sub-healthy type corresponding to the sub-healthy state.

[0032] The method described in the embodiment of the present application can provide a better business continuity plan for the enterprise by binding the association relationship, can ensure the continuous operation of the business when an unexpected failure occurs, and further improve the system stability and business continuity.

[0033] In a second aspect, the present application further provides a fault handling device, the device comprising:

[0034] A detection module, used to determine the communication status between the target node and the failed node when a failure is detected in any other node in the distributed cluster;

[0035] The processing module is used to determine the working status of the system disk on the faulty node when the communication status is normal, and to process the fault according to the working status; the working status is used to characterize the read and write capabilities and health status of the system disk; the processing includes isolation or fault remediation operations.

[0036] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0037] In case of detecting a failure of any other node in the distributed cluster, determining the communication status between the target node and the failed node;

[0038] When the communication status is normal, determine the working status of the system disk on the faulty node and handle the fault according to the working status; the working status is used to characterize the read and write capabilities and health status of the system disk; the processing includes isolation or fault remediation operations.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0040] In case of detecting a failure of any other node in the distributed cluster, determining the communication status between the target node and the failed node;

[0041] When the communication status is normal, determine the working status of the system disk on the faulty node and handle the fault according to the working status; the working status is used to characterize the read and write capabilities and health status of the system disk; the processing includes isolation or fault remediation operations.

[0042] In a fifth aspect, the present application further provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0043] In case of detecting a failure of any other node in the distributed cluster, determining the communication status between the target node and the failed node;

[0044] When the communication status is normal, determine the working status of the system disk on the faulty node and handle the fault according to the working status; the working status is used to characterize the read and write capabilities and health status of the system disk; the processing includes isolation or fault remediation operations.

[0045] The above fault handling method, device, computer equipment, storage medium and computer program product, the method determines the communication status between the target node and the faulty node when any other node in the distributed cluster is detected to have a fault, and determines the working status of the system disk on the faulty node when the communication status is normal, and handles the fault according to the working status. Among them, the working status is used to characterize the read and write capabilities and health status of the system disk, and the processing includes isolation or fault remediation operations. In the above method, when a fault occurs in the distributed cluster, the faulty node can be restored to normal use as much as possible by analyzing the working status on the faulty node and remediating the fault. Compared with the method of directly isolating the faulty node, the above fault remediation method can save the resource cost of the node. Moreover, for the faulty node that has been restored to normal after remediation, it can also reduce the human resources required for manual troubleshooting, thereby further reducing the cost of fault handling. In addition, the above method can detect from the outside to the inside in multiple dimensions, first detect the communication status between nodes, then detect the working function of the system disk in the node, and then detect the working status of the system disk, which can comprehensively and meticulously identify faults, not only improve the accuracy of fault detection, but also perform corresponding processing for different faults, thereby significantly improving the accuracy of fault handling. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of the structure of a distributed cluster in one embodiment;

[0047] Figure 2 A schematic diagram of a flow chart of a fault handling method in an embodiment;

[0048] Figure 3 is a flowchart of a fault handling method in another embodiment;

[0049] Figure 4 is a flowchart of a fault handling method in another embodiment;

[0050] Figure 5 is a flowchart of a fault handling method in another embodiment;

[0051] Figure 6 is a flowchart of a fault handling method in another embodiment;

[0052] Figure 7 is a flowchart of a fault handling method in another embodiment;

[0053] Figure 8 is a structural block diagram of a fault handling device in an embodiment;

[0054] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] A distributed storage system is composed of a server cluster consisting of multiple servers, and each server serves as a node and becomes the basis of the distributed storage system. When any node in the server cluster fails, the failed node needs to be processed quickly. If the failed node is not processed in time, it will affect the performance of the entire distributed storage system and even cause problems such as irreversible data loss. Therefore, it is very necessary to efficiently detect and process node failures. However, the processing method for node failure in the related art has the problem of high processing cost. The present application provides a fault handling method, which aims to solve the above technical problems. The following embodiments will specifically illustrate the fault handling method described in the present application.

[0057] The fault handling method provided in the embodiment of the present application can be applied to Figure 1 In the distributed cluster shown, the cluster includes a target node 101 and multiple other nodes 102, and each node is connected by communication. When a distributed cluster fails, a master node can be elected from each node, and the master node is used as the target node for fault processing. Each node includes a collection module 1011 and a processing module 1012, and the collection module 1011 and the processing module 1012 are connected by communication. When the target node 101 is determined, only the processing module 1012 of the target node 101 is effective, and the processing modules 1012 of other nodes are not effective. The collection module 1011 on each node is used to collect the status of each node. When a node fails, the collection module 1011 reports the fault information to the processing module 1012 on the target node 101, and the processing module 1012 on the target node 101 performs fault processing. The target node 101 can be, but not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. Optionally, the target node 101 can be implemented with an independent server or a server cluster composed of multiple servers. Other nodes 102 may be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. Optionally, other nodes 102 may be implemented using an independent server or a server cluster consisting of multiple servers.

[0058] Those skilled in the art will understand that Figure 1The structure shown in the figure is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the distributed cluster to which the solution of the present application is applied. The specific distributed cluster may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0059] In one embodiment, Figure 2 As shown, a fault handling method is provided, which is applied to Figure 1 The target node in the distributed cluster in is taken as an example to illustrate, including the following steps:

[0060] S201, when it is detected that any other node in the distributed cluster fails, determining the communication state between the target node and the failed node.

[0061] The communication status includes normal communication and abnormal communication.

[0062] In the embodiment of the present application, during the operation of each node in the distributed cluster, the management module of the target node can obtain the status information sent by the acquisition module on other nodes in real time, and then monitor each node according to the status. Optionally, each node can send the status information collected by the acquisition module on it to the management module of the target node regularly, and the target node can receive the status information of other nodes. After the target node obtains the status information sent by other nodes in the distributed cluster, it can analyze and detect the status information. If it is detected that any other node in the distributed cluster fails, the fault node can be diagnosed and the fault is handled. If it is detected that a fault occurs, the fault information can be reported to all other nodes in the cluster, and all other nodes will re-select the master, and the master node re-selected will be used as the target node, and the management module function on the master node will be effective, and then the target node will handle the fault of the fault node. During the operation of each node in the distributed cluster, the target node will periodically send heartbeat messages to all other nodes according to the heartbeat mechanism. When the target node detects that any other node in the distributed cluster fails, the communication status between the target node and the faulty node can be determined by the heartbeat message reply with the faulty node. Specifically, if the heartbeat message of the faulty node is not received within the preset time, it is determined that a communication failure occurs in the faulty node. The fault handling operation in this case is an isolation operation, that is, an isolation operation is performed on the faulty node.

[0063] S202: When the communication status is normal, determine the working status of the system disk on the faulty node, and process the fault according to the working status.

[0064] Among them, the working status is used to characterize the read and write capabilities and health status of the system disk. It can be understood that the read and write capabilities indicate whether the system disk can work, and the health status indicates whether the system disk can work efficiently; the processing includes isolation or fault remediation operations.

[0065] In the embodiment of the present application, during the operation of each node in the distributed cluster, the target node will send a working status query instruction to each other node in real time. After each node receives the status query instruction, it can return the working status to the target node. Specifically, when detecting the read and write capabilities of the system disk, the target node can determine whether the read and write capabilities of the system disk are abnormal based on whether the system disk of each node can return to the working status, or determine whether the read and write functions are abnormal based on the working status returned by the system disk of each node. For the health status of the system disk, the target node can send a linux mount command to each node, and after each node receives the command, the returned working status is whether the system disk is read-only; the target node can find the system disk mount partition and mount point through / etc / mtab, and use df + partition method to obtain the system disk usage and capacity, and the working status returned by each node is the current system disk capacity. The target node can periodically collect the iostat index (disk index) of each node system disk per second, and the working status returned by each node is the system disk type and the disk index of the system disk per second. The target node can detect whether the health status of the system disk is sub-healthy through the above query method. When the target node determines based on the above steps that the communication status between the target node and the faulty node is normal, in this case, it means that there is no communication failure between nodes in the distributed cluster. Then, the working status of the system disk on the faulty node can be further obtained through the above query method, and then the working capacity or health status of the system disk can be tested according to the working status, and the corresponding fault handling strategy can be executed according to different test results.

[0066] The fault handling method provided in the embodiment of the present application determines the communication status between the target node and the faulty node when any other node in the distributed cluster is detected to have a fault, and determines the working status of the system disk on the faulty node when the communication status is normal, and handles the fault according to the working status. Among them, the working status is used to characterize the read and write capabilities and health status of the system disk, and the processing includes isolation or fault remediation operations. In the above method, when a fault occurs in the distributed cluster, the faulty node can be restored to normal use as much as possible by analyzing the working status on the faulty node and remediating the fault. Compared with the method of directly isolating the faulty node, the above fault remediation method can save the resource cost of the node. Moreover, for the faulty node that has been restored to normal after remediation, it can also reduce the human resources required for manual troubleshooting, thereby further reducing the cost of fault handling. In addition, the above method can detect from the outside to the inside in multiple dimensions, first detect the communication status between nodes, then detect the working function of the system disk in the node, and then detect the working status of the system disk, which can comprehensively and meticulously identify faults, not only improve the accuracy of fault detection, but also perform corresponding processing for different faults, thereby significantly improving the accuracy of fault handling.

[0067] In one embodiment, a specific implementation method for handling a fault according to a working state is also provided, such as Figure 3 As shown, the "processing the fault according to the working status" in the above step S202 includes:

[0068] S301, determine whether the system disk on the faulty node has read / write abnormalities according to the working status; if the system disk has read / write abnormalities, execute S302; if the system disk does not have read / write abnormalities, execute S303.

[0069] In an embodiment of the present application, after obtaining the working status of the faulty node, the target node can determine whether the read and write capabilities of the system disk are abnormal based on whether the system disk of the faulty node can return to the working status, or can determine whether the read and write functions are abnormal based on the working status returned by the system disk of the faulty node. Specifically, if the system disk of the faulty node cannot return to the working status, determine whether there are read and write abnormalities on the system disk on the faulty node. If the system disk of the faulty node can return to the working status, further obtain the working status content returned by the faulty node based on preset instructions to determine whether there are read and write abnormalities. If the working status content returned by the faulty node is normal reading and writing, determine that there are no read and write abnormalities on the system disk on the faulty node. Otherwise, if the working status content returned by the faulty node is read and write abnormalities, determine that there are read and write abnormalities on the system disk on the faulty node.

[0070] S302, executing a first remedial operation; the first remedial operation includes restarting the faulty node, and processing the fault according to whether there is a read / write abnormality in the restarted faulty node.

[0071] The abnormal reading and writing situation includes abnormal reading and writing or normal reading and writing.

[0072] In an embodiment of the present application, if there is a read / write anomaly on the system disk, the first remedial operation is performed. Specifically, the target node can restart the faulty node, and then determine whether the read / write anomaly of the faulty node after the restart is eliminated. If the read / write anomaly of the faulty node after the restart is eliminated, it means that after the first remedial operation, the faulty node has returned to normal. If the read / write anomaly of the faulty node after the restart is not eliminated, it means that after the first remedial operation, the faulty node is still abnormal, and it can be further determined whether the number of restarts exceeds the preset number of restarts. If the number of restarts does not exceed the preset number of restarts, the next restart can be performed, and the above steps are repeated. If the number of restarts exceeds the preset number of restarts, the faulty node is isolated. The preset number of restarts can be determined according to business needs, and the preset number of restarts can be zero or non-zero.

[0073] S303: Determine the health status of the system disk on the faulty node according to the working status, and process the fault according to the health status.

[0074] Among them, health status includes normal state and sub-health state.

[0075] In the embodiment of the present application, if there is no read or write anomaly in the system disk, the health status of the system disk on the faulty node can be further determined based on the working status returned by the system disk. Specifically, it can be determined whether the health status of the faulty node is a sub-healthy state.

[0076] Specifically, the “processing the fault according to the health status” in the above step S303 includes:

[0077] When the health status indicates that the faulty node is in a sub-healthy state, the fault redundancy of the distributed cluster is obtained, and when it is determined that the fault redundancy is greater than a preset redundancy threshold, the fault is processed according to the sub-healthy type corresponding to the sub-healthy state.

[0078] Among them, the fault redundancy indicates the number of nodes that are allowed to fail in the distributed cluster. For example, if the fault redundancy is 3, it means that when the number of failed nodes in the distributed cluster is less than or equal to 3, it will not affect the normal operation of the entire cluster. In this case, no intervention is required on the failed nodes.

[0079] In an embodiment of the present application, a preset redundancy threshold of a distributed cluster can be determined in advance according to business needs. Then, when the target node determines that the health status indicates that the faulty node is in a sub-healthy state, the current fault redundancy of the distributed cluster can be obtained, and then it can be further determined whether the current fault redundancy is greater than the preset redundancy threshold. If the current fault redundancy is greater than the preset redundancy threshold, the sub-health type is further determined according to the working status, and then corresponding fault processing is performed according to different sub-health types; if the current fault redundancy is not greater than the preset redundancy threshold, no intervention is performed on the faulty node.

[0080] The method described in the embodiment of the present application divides the fault handling process into a read / write exception handling stage and a health status check stage, which can make fault handling more systematic, quickly locate problems, respond quickly, and reduce the time for fault recovery. Moreover, when judging the health status of the system disk, by analyzing the fault redundancy, it is possible to effectively avoid unnecessary migration or restart operations when resource usage has not reached the limit, thereby better utilizing cluster resources and saving costs.

[0081] In one embodiment, the above sub-health type is a read-only fault. On this basis, a specific implementation method for processing the fault according to the sub-health type corresponding to the sub-health state is also provided. The "processing the fault according to the sub-health type corresponding to the sub-health state" in the above steps includes:

[0082] Execute a second remedial operation; the second remedial operation includes restarting the faulty node and determining whether the read-only fault of the faulty node after the restart is eliminated. If not, determine whether the number of restarts exceeds the first preset number of restarts, and isolate the faulty node if the number of restarts exceeds the first preset number of restarts.

[0083] The read-only fault is that the system disk is in a read-only state. The first preset restart number can be determined according to business requirements, and the first preset restart number can be zero or non-zero. For example, for a business with a high requirement for fault response time, the first preset restart number can be set to a smaller value.

[0084] In an embodiment of the present application, when the target node receives a working status returned by the faulty node indicating that the system disk is in a read-only state, it can confirm that the sub-health type is a read-only fault. When the target node determines that a read-only fault occurs in the faulty node, a second remedial operation can be performed. Specifically, the target node can restart the faulty node, and then determine whether the read-only fault of the faulty node after the restart is eliminated. If the read-only fault of the faulty node after the restart is eliminated, it means that after the second remedial operation, the faulty node has returned to normal. If the read-only fault of the faulty node after the restart is not eliminated, it means that after the second remedial operation, the faulty node is still abnormal, and it can be further determined whether the number of restarts exceeds the first preset number of restarts. If the number of restarts does not exceed the first preset number of restarts, the next restart can be performed, and the above steps can be repeated. If the number of restarts exceeds the first preset number of restarts, the faulty node is isolated.

[0085] The method described in the embodiment of the present application sets an upper limit on the number of restarts to prevent excessive consumption of system resources due to unlimited restart operations, thereby ensuring the stability of the system during fault handling. Moreover, when restarting is invalid, the faulty node is isolated in time to avoid the spread of the fault, thereby protecting the health of the entire system and the security of the data.

[0086] In one embodiment, the above sub-health type is a capacity fault. On this basis, a specific implementation method for handling the fault according to the sub-health type corresponding to the sub-health state is also provided. The "handling the fault according to the sub-health type corresponding to the sub-health state" in the above steps includes:

[0087] Execute a third remedial operation; the third remedial operation includes clearing preset files under the faulty node, or prompting the user to clear the files; determine whether the capacity failure of the system disk of the faulty node after clearing is eliminated, and if not, isolate the faulty node.

[0088] Among them, capacity failure means that the capacity of the system disk exceeds the preset capacity threshold. The preset files include cache directories, logs and other files of the system software.

[0089] In an embodiment of the present application, after receiving the current capacity of the system disk returned by the faulty node, the target node can compare the current capacity of the system disk with the preset capacity threshold. If the current capacity of the system disk exceeds the preset capacity threshold, the sub-health type of the system disk is confirmed to be a capacity failure. When the target node determines that the faulty node has a capacity failure, a third remedial operation can be performed. Specifically, the target node can clear the preset files under the faulty node, or prompt the user to clear the operation, and then determine whether the capacity failure of the system disk of the faulty node after clearing is eliminated. If not, the faulty node is isolated. Optionally, clearing the preset files under the faulty node and prompting the user to clear the operation can be performed at the same time. Optionally, the preset files under the faulty node can be cleared first, and then a preliminary judgment can be made as to whether the capacity failure of the system disk of the faulty node after clearing is eliminated. If not, the user is prompted to clear the operation. Optionally, the user can be prompted to clear the operation first, and then a preliminary judgment can be made as to whether the capacity failure of the system disk of the faulty node after clearing is eliminated. If not, the preset files under the faulty node are cleared.

[0090] The method described in the embodiment of the present application can effectively release storage space by clearing preset files under the faulty node. At the same time, the user is prompted to perform the clearing operation in a timely manner, which can achieve the effect of human-computer interaction and improve the flexibility of fault handling.

[0091] In one embodiment, the above sub-health type is a slow disk fault. On this basis, a specific implementation method for handling the fault according to the sub-health type corresponding to the sub-health state is also provided. The "handling the fault according to the sub-health type corresponding to the sub-health state" in the above steps includes:

[0092] Execute the fourth remedial operation; the fourth remedial operation includes: determining whether the system disk on the faulty node is a disk array; if the system disk is a disk array, determining whether the faulty node needs to be isolated based on the number of abnormal alarms of the faulty node; if the system disk is not a disk array, isolating the faulty node.

[0093] Among them, the system disk is a disk array, which means that there are multiple disks in the system disk. When a disk has a slow disk failure, other disks may be in normal state. If the current system disk (disk) has a slow disk failure, you can restart it. After the restart, it may switch to a normal disk. If it switches to a normal disk, it means that the faulty node has been repaired.

[0094] In an embodiment of the present application, the target node can receive the system disk type and the disk index of the system disk per second returned by the faulty node, wherein the system disk type includes an HDD mechanical hard disk or an SSD solid-state disk, and the disk index is the wait value of the system disk (i.e., the average time for each IO). After receiving the system disk type and the disk index of the system disk per second returned by the faulty node, the target node can first determine the system disk type of the system disk, and then determine whether the disk index under the system disk type exceeds the preset index threshold. Different types of system disks correspond to different preset index thresholds, for example, Figure 4 As shown, the preset indicator threshold corresponding to the HDD mechanical hard disk can be 1000, and the preset indicator threshold corresponding to the SSD solid state disk can be 500. When the disk indicator exceeds the preset indicator threshold, the target node can determine that the sub-health type of the system disk is a slow disk failure. For example, when the HDD await value > 1000 and the SSD await value > 500, the sub-health type can be considered to be a slow disk failure. Optionally, for the accuracy of fault detection, the target node can perform fault analysis on the disk indicators within a preset time window. Specifically, the disk indicators within the preset time period can be first determined, and then the number of slow disk abnormalities that occur within the preset time period can be determined. If the number of slow disk abnormalities that occur within the preset time period is greater than the preset number threshold, the sub-health type of the system disk is considered to be a slow disk failure. Otherwise, it is considered that the system disk has not suffered a slow disk failure. For example, as Figure 4 As shown, if there are 20 slow disk abnormalities within 2 minutes, it is considered that the current system disk has a slow disk failure. When the target node determines that the faulty node has a slow disk failure, the fourth remedial operation can be performed. Specifically, the target node can first determine whether the system disk on the faulty node is a disk array, that is, determine whether the system disk uses RAID technology. If the system disk is a disk array, determine whether the faulty node needs to be isolated based on the number of abnormal alarms of the faulty node. If the system disk is not a disk array, isolate the faulty node.

[0095] Specifically, Figure 5 As shown, the above step of "determining whether to isolate the faulty node according to the number of abnormal alarms of the faulty node" includes:

[0096] S401, determine whether the number of abnormal alarms of the faulty node exceeds the preset number threshold, if the number of abnormal alarms of the faulty node does not exceed the preset number threshold, execute S402, if the number of abnormal alarms of the faulty node exceeds the preset number threshold, execute S403.

[0097] Among them, the number of abnormal alarms is the number of slow disk failure alarms.

[0098] In the embodiment of the present application, after determining that the system disk is a disk array, the target node may compare the number of abnormal alarms with a preset number threshold to determine whether the number of abnormal alarms of the system disk exceeds the preset number threshold.

[0099] S402, restart the faulty node and determine whether the read-only fault of the faulty node after the restart is eliminated. If not eliminated, determine whether the number of restarts exceeds a second preset number of restarts, and isolate the faulty node if the number of restarts exceeds the second preset number of restarts.

[0100] The preset number of times threshold can be determined according to business requirements. The second preset number of restart times can be determined according to business requirements, and can be zero or non-zero.

[0101] In an embodiment of the present application, if the number of abnormal alarms of the faulty node does not exceed the preset number threshold, the faulty node is restarted, and it is determined whether the read-only fault of the faulty node after the restart is eliminated. If not, it is determined whether the number of restarts exceeds a second preset restart number, and if the number of restarts exceeds the second preset restart number, the faulty node is isolated.

[0102] S403: Isolate the faulty node.

[0103] In the embodiment of the present application, if the number of abnormal alarms of the faulty node exceeds a preset threshold, the faulty node is isolated.

[0104] The method described in the embodiment of the present application can perform more targeted management and optimize resource usage by judging whether the system disk on the faulty node is a disk array. In addition, by analyzing the number of abnormal alarms when the system disk is a disk array, data support can be provided for preventive maintenance of the system, reducing the risk of sudden failures and further improving the accuracy of fault handling.

[0105] In one example, if Figure 6 As shown, when it is determined that the fault redundancy is greater than the preset redundancy threshold, the fault handling method further includes:

[0106] S501, determining whether to allow isolation of the faulty node according to the operation status of the target service module when the faulty node fails.

[0107] The target service module is a service module associated with the fault.

[0108] In the embodiment of the present application, the association relationship between each service module can be bound in advance. When the target node determines that the fault redundancy is greater than the preset redundancy threshold, the operation of the target service module when the fault node fails can be analyzed to determine whether the fault node can be isolated. If not allowed, after a preset time interval, it is determined again whether the fault node can be isolated. If the fault node is still not allowed to be isolated after reaching the preset number of times, the abnormal situation is reported or forced isolation is performed.

[0109] S502, when it is allowed to isolate the faulty node, the process returns to the step of processing the fault according to the sub-health type corresponding to the sub-health state.

[0110] In the embodiment of the present application, when isolation of the faulty node is allowed, the process returns to execute the step of processing the fault according to the sub-health type corresponding to the sub-health state.

[0111] The method described in the embodiment of the present application can provide a better business continuity plan for the enterprise by binding the association relationship, can ensure the continuous operation of the business when an unexpected failure occurs, and further improve the system stability and business continuity.

[0112] Based on all the above embodiments, a fault handling method is also provided. Figure 7 As shown, the method includes:

[0113] S601, when it is detected that any other node in the distributed cluster fails, determine the communication state between the target node and the failed node.

[0114] S602: When the communication status is normal, determine the working status of the system disk on the faulty node, and determine whether the system disk on the faulty node has read / write abnormalities according to the working status. The working status is used to characterize the read / write capability and health status of the system disk.

[0115] S603: If the system disk has a read / write abnormality, a first remedial operation is performed, wherein the first remedial operation includes restarting the faulty node and processing the fault according to whether the restarted faulty node has a read / write abnormality.

[0116] S604: If there is no reading or writing abnormality in the system disk, determine the health status of the system disk on the faulty node according to the working status.

[0117] S605: When the health status indicates that the faulty node is in a sub-healthy state, obtain the fault redundancy of the distributed cluster.

[0118] S606: When it is determined that the fault redundancy is greater than the preset redundancy threshold, determine whether to allow the faulty node to be isolated according to the operation of the target service module when the faulty node fails. The target service module is a service module associated with the fault.

[0119] S607: When the faulty node is allowed to be isolated, the fault is processed according to the sub-health type corresponding to the sub-health state.

[0120] S608, the sub-health type is a read-only fault, and a second remedial operation is performed. The second remedial operation includes restarting the faulty node and determining whether the read-only fault of the restarted faulty node is eliminated. If not eliminated, determining whether the number of restarts exceeds a first preset number of restarts, and isolating the faulty node if the number of restarts exceeds the first preset number of restarts.

[0121] S609, the sub-health type is capacity failure, and the third remedial operation is performed. The third remedial operation includes clearing the preset files under the faulty node, or prompting the user to clear the files; determining whether the capacity failure of the system disk of the faulty node after clearing is eliminated, and if not, isolating the faulty node.

[0122] S610, the sub-health type is a slow disk fault, and the fourth remedial operation is performed. The fourth remedial operation includes determining whether the system disk on the faulty node is a disk array; if the system disk is a disk array, determining whether the number of abnormal alarms of the faulty node exceeds a preset number threshold; if the number of abnormal alarms of the faulty node does not exceed the preset number threshold, restarting the faulty node and determining whether the read-only fault of the faulty node after the restart is eliminated, if not eliminated, determining whether the number of restarts exceeds a second preset number of restarts, and isolating the faulty node if the number of restarts exceeds the second preset number of restarts; isolating the faulty node if the number of abnormal alarms of the faulty node exceeds the preset number threshold; if the system disk is not a disk array, isolating the faulty node.

[0123] The methods described in the above steps are all described in the above embodiments. Please refer to the above description for details and will not be repeated here.

[0124] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0125] Based on the same inventive concept, the embodiment of the present application also provides a fault handling device for implementing the fault handling method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more fault handling device embodiments provided below can refer to the limitations on the fault handling method above, and will not be repeated here.

[0126] In one embodiment, Figure 8 As shown, a fault handling device is provided, comprising:

[0127] The detection module 11 is used to determine the communication status between the target node and the failed node when a failure is detected in any other node in the distributed cluster.

[0128] The processing module 12 is used to determine the working status of the system disk on the faulty node when the communication status is normal, and to process the fault according to the working status; the working status is used to characterize the read and write capabilities and health status of the system disk; the processing includes isolation or fault remediation operations.

[0129] In one embodiment, the processing module 12 includes:

[0130] The first determining unit is used to determine whether there is a read / write abnormality in the system disk on the faulty node according to the working status.

[0131] The first execution unit is used to execute a first remedial operation if there is a read / write abnormality in the system disk; the first remedial operation includes restarting the faulty node, and processing the fault according to whether there is a read / write abnormality in the restarted faulty node.

[0132] The second execution unit is used to determine the health status of the system disk on the faulty node according to the working status if there is no read or write abnormality in the system disk, and process the fault according to the health status.

[0133] In one embodiment, the above-mentioned second execution unit is specifically used to obtain the fault redundancy of the distributed cluster when the health status indicates that the fault node is in a sub-healthy state, and when it is determined that the fault redundancy is greater than a preset redundancy threshold, process the fault according to the sub-health type corresponding to the sub-healthy state.

[0134] In one embodiment, the above-mentioned second execution unit is specifically used to perform a second remedial operation; the second remedial operation includes restarting the faulty node and determining whether the read-only fault of the faulty node after the restart is eliminated. If not, determining whether the number of restarts exceeds a first preset number of restarts, and isolating the faulty node if the number of restarts exceeds the first preset number of restarts.

[0135] In one embodiment, the above-mentioned second execution unit is specifically used to execute a third remedial operation; the third remedial operation includes clearing the preset files under the faulty node, or prompting the user to perform a clearing operation; determining whether the capacity fault of the system disk of the faulty node is eliminated after clearing, and if not, isolating the faulty node.

[0136] In one embodiment, the above-mentioned second execution unit is specifically used to perform a fourth remedial operation; the fourth remedial operation includes: determining whether the system disk on the faulty node is a disk array; if the system disk is a disk array, determining whether the faulty node needs to be isolated based on the number of abnormal alarms of the faulty node; if the system disk is not a disk array, isolating the faulty node.

[0137] In one embodiment, the second execution unit is specifically used to determine whether the number of abnormal alarms of the faulty node exceeds a preset number threshold;

[0138] If the number of abnormal alarms of the faulty node does not exceed the preset number threshold, the faulty node is restarted, and it is determined whether the read-only fault of the faulty node after the restart is eliminated. If not eliminated, it is determined whether the number of restarts exceeds a second preset number of restarts, and if the number of restarts exceeds the second preset number of restarts, the faulty node is isolated;

[0139] If the number of abnormal alarms of a faulty node exceeds a preset threshold, the faulty node will be isolated.

[0140] In one embodiment, the second execution unit is specifically used to determine whether to allow the faulty node to be isolated according to the operation status of the target service module when the faulty node fails; the target service module is a service module associated with the fault;

[0141] In the case where the faulty node is allowed to be isolated, the process returns to the step of processing the fault according to the sub-healthy type corresponding to the sub-healthy state.

[0142] Each module in the above fault handling device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0143] In one embodiment, a computer device is provided. The computer device may be a terminal or a server. The internal structure diagram thereof may be as follows: Fig. 9 As shown, the computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a fault handling method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0144] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0145] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the fault handling method described in any of the above embodiments is implemented.

[0146] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the fault handling method described in any of the above embodiments is implemented.

[0147] In one embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the fault handling method described in any of the above embodiments is implemented.

[0148] The above embodiment provides a computer program product, whose implementation principle and technical effect are similar to those of the above method embodiment, and will not be repeated here.

[0149] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0150] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A fault handling method, characterized in that: Applied to a target node in a distributed cluster, the method comprises: In case a failure of any other node in the distributed cluster is detected, determining a communication state between the target node and the failed node; When the communication status is normal, determine the working status of the system disk on the faulty node, and handle the fault according to the working status; the working status is used to characterize the read and write capabilities and health status of the system disk; the processing includes isolation or fault remediation operations.

2. The method according to claim 1, characterized in that The processing of the fault according to the working state includes: Determine whether there is a read / write abnormality on the system disk of the faulty node according to the working status; If the system disk has a read / write anomaly, a first remedial operation is performed; the first remedial operation includes restarting the faulty node, and processing the fault according to whether the restarted faulty node has the read / write anomaly; If there is no reading or writing abnormality in the system disk, the health status of the system disk on the faulty node is determined according to the working status, and the fault is processed according to the health status.

3. The method according to claim 2, characterized in that The processing of the fault according to the health status includes: When the health status indicates that the faulty node is in a sub-healthy state, the fault redundancy of the distributed cluster is obtained, and when it is determined that the fault redundancy is greater than a preset redundancy threshold, the fault is processed according to the sub-healthy type corresponding to the sub-healthy state.

4. The method according to claim 3, characterized in that The sub-health type is a read-only fault, and the processing of the fault according to the sub-health type corresponding to the health state includes: Execute a second remedial operation; the second remedial operation includes restarting the faulty node and determining whether the read-only fault of the faulty node after the restart is eliminated. If not, determine whether the number of restarts exceeds the first preset number of restarts, and isolate the faulty node if the number of restarts exceeds the first preset number of restarts.

5. The method according to claim 3, characterized in that: The sub-health type is a capacity fault, and the processing of the fault according to the sub-health type corresponding to the sub-health state includes: Execute a third remedial operation; the third remedial operation includes clearing preset files under the faulty node, or prompting the user to perform a clearing operation; determine whether the capacity failure of the system disk of the faulty node after clearing is eliminated, and if not, isolate the faulty node.

6. The method according to claim 3, characterized in that The sub-health type is a slow disk fault, and the processing of the fault according to the sub-health type corresponding to the sub-health state includes: Execute a fourth remedial operation; the fourth remedial operation includes: determining whether the system disk on the faulty node is a disk array; if the system disk is the disk array, determining whether the faulty node needs to be isolated based on the number of abnormal alarms of the faulty node; if the system disk is not the disk array, isolating the faulty node.

7. The method according to claim 6, characterized in that The determining whether the faulty node needs to be isolated according to the number of abnormal alarms of the faulty node includes: Determine whether the number of abnormal alarms of the faulty node exceeds a preset number threshold; If the number of abnormal alarms of the faulty node does not exceed the preset number threshold, the faulty node is restarted, and it is determined whether the read-only fault of the restarted faulty node is eliminated. If not eliminated, it is determined whether the number of restarts exceeds a second preset number of restarts, and if the number of restarts exceeds the second preset number of restarts, the faulty node is isolated; If the number of abnormal alarms of the faulty node exceeds a preset threshold, the faulty node is isolated.

8. The method according to any one of claims 3 to 7, characterized in that: The method further comprises: Determine whether to allow isolation of the faulty node according to the operation status of the target service module when the faulty node fails; the target service module is a service module associated with the fault; In the case where the faulty node is allowed to be isolated, the process returns to the step of processing the fault according to the sub-healthy type corresponding to the sub-healthy state.

9. A fault handling device, characterized in that: The device comprises: A detection module, used to determine the communication status between the target node and the failed node when a failure is detected in any other node in the distributed cluster; A processing module is used to determine the working status of the system disk on the faulty node when the communication status is normal, and process the fault according to the working status; the working status is used to characterize the read and write capabilities and health status of the system disk; the processing includes isolation or fault remediation operations.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images