Fault processing method and device, computer equipment, storage medium and program product
By using preset processing strategies in distributed storage systems, the problem of low fault processing efficiency under traditional methods is solved, efficient and accurate fault processing is achieved, and the high availability of the system is improved.
Patent Information
- Application Number
- CN202411515086.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-01-14
AI Technical Summary
Traditional fault handling methods When multiple server nodes in a distributed storage system have multiple types of faults, the processing efficiency is low and manual intervention is required, resulting in a degradation of system performance.
Preset processing strategies are used to determine the fault type and handle it accordingly, including node faults and disk fault handling policies, and automatically call the corresponding policies for fault handling to improve fault handling efficiency and accuracy.
Without manual intervention, it can accurately handle the failures of multiple types of fault nodes in distributed storage systems, improve fault processing efficiency, and improve the high availability and performance of the system.
Smart Images
Figure CN119311459A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of distributed storage technology, and in particular to a fault handling method, apparatus, computer equipment, storage medium and program product. Background Art
[0002] The distributed storage system includes multiple server nodes, and each server node can be equipped with multiple hard disks. During the operation of the distributed storage system, node pseudo-death, slow disk and other faults may occur. Node pseudo-death means that the network between the faulty node and other nodes is not interrupted, but the faulty node cannot provide services; slow disk means that the data processing rate of the faulty hard disk is slower than that of other hard disks in the server cluster. Therefore, in order to ensure the normal use of the distributed storage system, the above faults need to be handled in a timely manner.
[0003] However, for the scenario where multiple server nodes in a distributed storage system have multiple types of faults at the same time, traditional fault handling methods have the problem of low processing efficiency. Summary of the invention
[0004] Based on this, it is necessary to provide a fault handling method, device, computer equipment, storage medium and program product to improve the efficiency of fault handling in the scenario where multiple types of faults exist simultaneously on multiple server nodes in a distributed storage system in response to the above technical problems.
[0005] In a first aspect, the present application provides a fault handling method, which is applied to a distributed storage system, comprising:
[0006] Determining the fault type of each faulty node in the distributed storage system;
[0007] When there are faulty nodes of different fault types in the distributed storage system, a preset processing strategy is used to perform fault processing on each of the faulty nodes; the preset processing strategy includes processing strategies for different types of faults.
[0008] In the above fault handling method, the fault type of each faulty node in the distributed storage system is determined; when there are faulty nodes of different fault types in the distributed storage system, a preset processing strategy is used to handle the faults of each faulty node; the preset processing strategy includes processing strategies for different types of faults. Since the preset processing strategy includes processing strategies for different types of faults, for the scenario where multiple server nodes in the distributed storage system have multiple types of faults at the same time, the embodiment of the present application can use the preset processing strategy to accurately handle the faults of faulty nodes of various fault types without the need for manual intervention for fault handling, thereby improving the fault handling efficiency of the distributed storage system.
[0009] In one embodiment, the preset processing strategy includes a node failure processing strategy and a disk failure processing strategy, and the preset processing strategy is used to perform failure processing on each of the failed nodes, including:
[0010] For each of the faulty nodes, if the fault type of the faulty node is a node fault type, the node fault handling strategy is called to handle the fault of the faulty node; if the fault type of the faulty node is a disk fault type, the disk fault handling strategy is called to handle the fault of the faulty node.
[0011] The method of this embodiment is for each faulty node. If the fault type of the faulty node is a node fault type, the node fault handling strategy is called to handle the faulty node; if the fault type of the faulty node is a disk fault type, the disk fault handling strategy is called to handle the faulty node. In this way, according to the fault type of each faulty node, the fault handling strategy corresponding to the fault type can be automatically called to handle the fault, thereby improving the fault handling efficiency and accuracy of the distributed storage system.
[0012] In one of the embodiments, calling the node fault handling strategy to handle the fault of the faulty node includes:
[0013] Sending a restart instruction to the faulty node; the restart instruction is used to instruct the faulty node to perform a restart operation;
[0014] Alternatively, the service interaction with the faulty node is disconnected.
[0015] The method of this embodiment can send a restart instruction to the faulty node, wherein the restart instruction is used to instruct the faulty node to perform a restart operation; or disconnect the service interaction with the faulty node. In this way, the fault of the faulty node can be effectively handled by restarting the faulty node or performing service avoidance on the faulty node.
[0016] In one embodiment, calling the disk fault handling strategy to handle the fault of the faulty node includes:
[0017] Determine whether there is still a node failure on the faulty node;
[0018] If the node fault still exists on the faulty node, node fault processing is performed on the faulty node to obtain a processed node, and disk fault processing is performed on the processed node;
[0019] If the node failure does not exist on the faulty node, disk failure processing is performed on the faulty node.
[0020] The method of this embodiment can determine the corresponding fault handling method according to whether there are other node faults on the faulty node. Thus, different fault handling methods can be used to correspondingly handle faulty nodes containing different fault types, thereby improving the accuracy of fault handling in the distributed storage system.
[0021] In one embodiment, the performing disk failure processing on the failed node includes:
[0022] A disk isolation instruction is sent to the faulty node; the disk isolation instruction is used to instruct the faulty node to isolate the faulty disk.
[0023] The method of this embodiment can instruct the faulty node to isolate the faulty disk by sending a disk isolation instruction to the faulty node, so that the faulty disk will not affect the service in the corresponding faulty node.
[0024] In one embodiment, determining the fault type of each faulty node in the distributed storage system includes:
[0025] Sending a fault test request to each node in the distributed storage system;
[0026] According to the response message corresponding to the fault test request fed back by each of the nodes, the faulty nodes in the distributed storage system and the fault type of each of the faulty nodes are determined.
[0027] The method of this embodiment can send a fault test request to each node in the distributed storage system, so that each node can perform a fault test on other nodes in the distributed storage system based on the fault test request, and feed back the fault test result to the management node in a response message corresponding to the fault test request. Then, the management node can accurately determine the faulty nodes in the distributed storage system and the fault type of each faulty node based on the response message corresponding to the fault test request fed back by each node.
[0028] In a second aspect, the present application further provides a fault handling device, which is applied to a distributed storage system, and the device includes:
[0029] A determination module, used to determine the fault type of each faulty node in the distributed storage system;
[0030] The fault processing module is used to adopt a preset processing strategy to perform fault processing on each fault node when there are fault nodes of different fault types in the distributed storage system; the preset processing strategy includes processing strategies for different types of faults.
[0031] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method in any one of the embodiments of the first aspect are implemented.
[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method in any one of the embodiments of the first aspect are implemented.
[0033] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method in any one of the embodiments of the first aspect are implemented.
[0034] The above-mentioned fault handling method, device, computer equipment, storage medium and program product determine the fault type of each faulty node in the distributed storage system; when there are faulty nodes of different fault types in the distributed storage system, a preset processing strategy is used to perform fault handling on each faulty node; the preset processing strategy includes processing strategies for different types of faults. Since the preset processing strategy includes processing strategies for different types of faults, for the scenario where multiple server nodes in the distributed storage system have multiple types of faults at the same time, the embodiments of the present application can use the preset processing strategy to accurately perform fault handling on faulty nodes of various fault types without the need for manual intervention for fault handling, thereby improving the fault handling efficiency of the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0036] Figure 1 An application environment diagram of a fault handling method in an embodiment;
[0037] Figure 2 A schematic diagram of a flow chart of a fault handling method in an embodiment;
[0038] Figure 3 A schematic diagram of a process flow of a fault handling step in an embodiment;
[0039] Figure 4 A flowchart of disk failure processing steps in one embodiment;
[0040] Figure 5 is a flowchart of a fault handling method in another embodiment;
[0041] Figure 6 is a structural block diagram of a fault handling device in an embodiment;
[0042] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.
[0045] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.
[0046] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0047] The distributed storage system includes multiple server nodes, and each server node can be equipped with multiple hard disks. During the operation of the distributed storage system, node pseudo-death, slow disk and other faults may occur. Node pseudo-death means that the network between the faulty node and other nodes is not interrupted, but the faulty node cannot provide services; slow disk means that the data processing rate of the faulty hard disk is slower than that of other hard disks in the server cluster. For the distributed storage system, if the fault is not handled in time, the performance of the entire distributed storage system will be degraded. Therefore, in order to ensure the normal use of the distributed storage system, the above faults need to be handled in time.
[0048] However, for the scenario where multiple server nodes in a distributed storage system have multiple types of faults at the same time, the traditional fault handling method can only handle the faults through manual intervention. Therefore, the traditional fault handling method has the problem of low processing efficiency.
[0049] After introducing the background technology of the fault handling method provided by the embodiment of the present application, the implementation environment involved in the fault handling method provided by the embodiment of the present application will be briefly described below. The fault handling method provided by the embodiment of the present application can be applied to Figure 1 In the distributed storage system shown. The distributed storage system includes multiple server nodes 101 and multiple disk pools 102, each disk pool 102 includes multiple disks 103, and each server node 101 can be provided with multiple disks 103, and each server node 101 can communicate with each other, perform data transmission and information exchange and other operations, and each node and each disk can perform tasks in parallel. Among them, the server node 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Exemplarily, each physical server can be, but not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.
[0050] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the distributed storage system to which the scheme of the present application is applied. The specific distributed storage system may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0051] In one embodiment, Figure 2 As shown, a fault handling method is provided, which is applied to Figure 1 The application in the distributed storage system is taken as an example to illustrate, including the following steps:
[0052] S201, determining the fault type of each faulty node in the distributed storage system.
[0053] Among them, a faulty node refers to a node that fails in a distributed storage system. The fault type may include but is not limited to at least one of a node fault type, a disk fault type, a service fault type, etc. The node fault type refers to the type of node pseudo-death fault that occurs in the node, the disk fault type refers to the type of slow disk fault that occurs in the disk, and the service fault type refers to the type of service pseudo-death fault that occurs in the service in the node.
[0054] The embodiments of the present application involve two ways of determining the fault type of each faulty node in a distributed storage system. The first way is that each node in the distributed storage system can actively perform fault detection on other nodes in the distributed storage system, determine the faulty nodes in the distributed storage system and the fault types of each faulty node, and report the faulty nodes in the distributed storage system and the fault types of each faulty node to the management node in the distributed storage system in real time or periodically; the second way is that the distributed storage system can notify each node in the distributed storage system to perform a fault test, so that each node that receives the notification performs fault detection on other nodes in the distributed storage system, determines the faulty nodes in the distributed storage system and the fault types of each faulty node, and returns the faulty nodes in the distributed storage system and the fault types of each faulty node to the management node in the distributed storage system. Of course, the embodiments of the present application do not limit the specific implementation method for determining the fault type of each faulty node. Among them, the management node refers to the node used for fault processing in the distributed storage system.
[0055] S202, when there are faulty nodes of different fault types in the distributed storage system, a preset processing strategy is used to process the faults of the faulty nodes; the preset processing strategy includes processing strategies for different types of faults.
[0056] Among them, the preset processing strategy includes processing strategies for different types of failures. For example, the preset processing strategy may include but is not limited to the processing strategy corresponding to the node failure type, the processing strategy corresponding to the disk failure type, the processing strategy corresponding to the service failure type, etc., and the processing strategy corresponding to the node failure type, the processing strategy corresponding to the disk failure type and the processing strategy corresponding to the service failure type are all different.
[0057] In an embodiment of the present application, after determining the fault type of each faulty node in the distributed storage system, the management node in the distributed storage system can determine whether the fault type of each faulty node is the same, and can determine whether there are faulty nodes with different fault types in the distributed storage system. When there are faulty nodes with different fault types in the distributed storage system, the management node in the distributed storage system can use a preset processing strategy to perform fault processing on each faulty node. Exemplarily, the management node in the distributed storage system can use the preset processing strategy corresponding to the fault type of each node to perform fault processing on the faulty nodes corresponding to the fault type of each node in parallel; or, the management node in the distributed storage system can also determine the processing priority of the preset processing strategy corresponding to the fault type of each node, and according to the above processing priority, use the preset processing strategy corresponding to the fault type of each node to perform fault processing on the faulty nodes corresponding to the fault type of each node. Of course, the embodiment of the present application does not limit the specific implementation method of fault processing.
[0058] In the above fault handling method, the fault type of each faulty node in the distributed storage system is determined; when there are faulty nodes of different fault types in the distributed storage system, a preset processing strategy is used to handle the faults of each faulty node; the preset processing strategy includes processing strategies for different types of faults. Since the preset processing strategy includes processing strategies for different types of faults, for the scenario where multiple server nodes in the distributed storage system have multiple types of faults at the same time, the embodiment of the present application can use the preset processing strategy to accurately handle the faults of faulty nodes of various fault types without the need for manual intervention for fault handling, thereby improving the fault handling efficiency of the distributed storage system.
[0059] In one embodiment, the above-mentioned preset processing strategy includes a node fault processing strategy and a disk fault processing strategy. Based on this, a fault processing implementation method is provided, namely, "using the preset processing strategy to perform fault processing on each faulty node" in the above-mentioned S202. Figure 3 As shown, including:
[0060] S301, for each faulty node, determine whether the fault type of the faulty node is a node fault type; if the fault type of the faulty node is a node fault type, execute S302; if the fault type of the faulty node is not a node fault type but a disk fault type, execute S303.
[0061] S302: Calling a node fault handling strategy to handle a faulty node.
[0062] S303: Call the disk fault handling strategy to handle the fault of the faulty node.
[0063] The node failure type refers to the type of node pseudo-death failure that occurs on the node, and the disk failure type refers to the type of slow disk failure that occurs on the disk in the node.
[0064] In the embodiment of the present application, since the above-mentioned preset processing strategy includes a node fault processing strategy and a disk fault processing strategy, for each faulty node, the management node in the distributed storage system can determine whether the fault type of the faulty node is a node fault type. If it is determined that the fault type of the faulty node is a node fault type, the management node in the distributed storage system can directly call the node fault processing strategy to perform fault processing on the faulty node. At this time, regardless of whether there is a disk fault type fault (referred to as disk fault) on each faulty node in the distributed storage system, the node fault on the faulty node determined is first processed. After the node fault processing is completed, the management node can process the disk fault on each faulty node. If it is determined that the fault type of the faulty node is not a node fault type, but a disk fault type, the management node in the distributed storage system can call the disk fault processing strategy to perform fault processing on the faulty node. Optionally, the management node can directly call the disk fault processing strategy to perform fault processing on the faulty disk in the faulty node; or, the management node can also determine the processing priority in the disk fault processing strategy, and perform fault processing on the disk fault and node fault in the faulty node according to the processing priority. Of course, the embodiment of the present application does not limit the specific implementation method of calling the disk fault processing strategy for fault processing.
[0065] The method of this embodiment is for each faulty node. If the fault type of the faulty node is a node fault type, the node fault handling strategy is called to handle the faulty node; if the fault type of the faulty node is a disk fault type, the disk fault handling strategy is called to handle the faulty node. In this way, according to the fault type of each faulty node, the fault handling strategy corresponding to the fault type can be automatically called to handle the fault, thereby improving the fault handling efficiency and accuracy of the distributed storage system.
[0066] In one embodiment, an implementation method for invoking a node fault handling strategy to handle a fault is provided, namely, “invoking a node fault handling strategy to handle a faulty node” in the above S302, including:
[0067] Send a restart instruction to the failed node; the restart instruction is used to instruct the failed node to perform a restart operation; or, disconnect the business interaction with the failed node.
[0068] The restart instruction is used to instruct the failed node to perform a restart operation, and the business interaction refers to the communication connection or service interaction between multiple nodes in the distributed storage system.
[0069] In the embodiment of the present application, the method of calling the node fault handling strategy to handle the fault of the faulty node includes the following two methods: the first method is that the management node in the distributed storage system can send a restart instruction to the faulty node, so that after receiving the restart instruction, the faulty node can perform a restart operation according to the restart instruction. The second method is that the management node in the distributed storage system can directly disconnect the business interaction with the faulty node to avoid the business of the faulty node at the code level, and then the faulty node will not be used to implement specific business.
[0070] The method of this embodiment can send a restart instruction to the faulty node, wherein the restart instruction is used to instruct the faulty node to perform a restart operation; or disconnect the service interaction with the faulty node. In this way, the fault of the faulty node can be effectively handled by restarting the faulty node or performing service avoidance on the faulty node.
[0071] In one embodiment, a method for implementing calling a disk fault handling strategy to handle a fault is provided, namely, "calling a disk fault handling strategy to handle a faulty node" in the above S303, such as Figure 4 As shown, including:
[0072] S401, determine whether there is still a node failure on the faulty node; if there is still a node failure on the faulty node, execute S402; if there is no node failure on the faulty node, execute S403.
[0073] S402, performing node failure processing on the failed node to obtain a processed node, and performing disk failure processing on the processed node.
[0074] S403: Perform disk failure processing on the failed node.
[0075] The node failure refers to a node failure type failure.
[0076] In the embodiment of the present application, when it is determined that the fault type of the faulty node is a disk fault type, the management node in the distributed storage system can determine whether there is still a node fault on the faulty node. If it is determined that there is still a node fault on the faulty node, it means that there are both disk faults and node faults on the faulty node. Then, the detected disk fault may be a misjudgment caused by the node fault. Therefore, in order to avoid misjudgment of disk faults and triggering operations such as disk replacement or kicking, the management node in the distributed storage system can return or clear the mark corresponding to the disk fault, first perform node fault processing on the faulty node, obtain the processed node, and after the node fault processing is completed, the management node can re-determine the faulty disk in the processed node, and perform disk fault processing on the faulty disk in the processed node. If it is determined that there is no node fault on the faulty node, it means that there is only a disk fault on the faulty node. At this time, even if there are node faults in other faulty nodes in the distributed storage system, the management node in the distributed storage system can directly prioritize disk fault processing on the faulty node. After the disk fault processing of the faulty node is completed, node fault processing can be performed on other faulty nodes.
[0077] The method of this embodiment can determine the corresponding fault handling method according to whether there are other node faults on the faulty node. Thus, different fault handling methods can be used to correspondingly handle faulty nodes containing different fault types, thereby improving the accuracy of fault handling in the distributed storage system.
[0078] In one embodiment, a method for implementing disk failure processing is provided, namely, the “processing a disk failure on a failed node” in S403, including:
[0079] A disk isolation instruction is sent to the faulty node; the disk isolation instruction is used to instruct the faulty node to isolate the faulty disk.
[0080] In an embodiment of the present application, a management node in a distributed storage system can send a disk isolation instruction to a faulty node, so that after receiving the disk isolation instruction, the faulty node can determine at least one faulty disk in the faulty node according to the identifier of the faulty disk in the faulty node carried in the disk isolation instruction, and isolate or kick out at least one faulty disk from the faulty node, and store the data in the faulty disk in other disks in the distributed storage system. The disk isolation instruction is used to instruct the faulty node to isolate the faulty disk, and the disk isolation instruction includes the identifier of the faulty disk in the faulty node.
[0081] The method of this embodiment can instruct the faulty node to isolate the faulty disk by sending a disk isolation instruction to the faulty node, so that the faulty disk will not affect the service in the corresponding faulty node.
[0082] In one embodiment, a method for determining the fault type of each faulty node is provided, that is, the “determining the fault type of each faulty node in the distributed storage system” in S201 above, including:
[0083] Send a fault test request to each node in the distributed storage system; determine the faulty nodes and the fault types of each faulty node in the distributed storage system according to the response messages corresponding to the fault test requests fed back by each node.
[0084] In an embodiment of the present application, a management node in a distributed storage system may send a fault test request to each node in the distributed storage system to notify each node in the distributed storage system to perform a fault test, whereby each node that receives the notification may perform a fault test on other nodes in the distributed storage system based on the fault test request, thereby determining the faulty nodes in the distributed storage system and the fault type of each faulty node, and carrying the faulty nodes in the distributed storage system and the fault type of each faulty node in a response message corresponding to the fault test request, and then sending the response message corresponding to the fault test request to the management node in the distributed storage system, and then the management node may receive the response message corresponding to the fault test request fed back by each node in the distributed storage system, and determine the faulty nodes in the distributed storage system and the fault type of each faulty node according to the information carried in the response message fed back by each node.
[0085] Exemplarily, if the first node in the distributed storage system sends a node fault test message to the second node in the distributed storage system, but the first node does not receive a response message corresponding to the node fault test message returned by the second node within a preset time period, it means that the second node has a node pseudo-death fault. At this time, the first node can determine the second node as a faulty node, and determine the fault type of the second node as a node fault type. After that, the first node can report the above fault information to the main service oJmgs in the management node, so that the main service oJmgs in the management node can add a node fault mark to the second node, so that the underlying services can obtain the fault type of the second node. Among them, the first node and the second node are any two nodes in the distributed storage system. For example, the node fault mark can be node-failure=true.
[0086] Exemplarily, after the degradation check passes (i.e., it has been confirmed in advance that there are no faults other than node failures and disk failures in the distributed storage system), the slave service mgcd where the first disk in the distributed storage system is located can obtain the time consumption information of the IO operation in the first disk, and compare the time consumption information with the preset time consumption threshold. If the time consumption information is greater than the preset time consumption threshold, the slave service mgcd where the first disk is located can report to the master service oJmgs in the management node that there is a disk failure in the first disk, so that the master service oJmgs in the management node can add a disk failure mark to the first disk. The first disk is any disk in the distributed storage system, oJmgs is a management service in the Parastor system, a master-slave architecture is adopted, mgcd is a downstream service of oJmgs, and the slave service mgcd can communicate directly with the master service oJmgs. For example, the disk failure mark can be is-slow=true.
[0087] In addition, the redundancy of the cluster where the nodes and disks are located needs to be considered to ensure that the determined faulty nodes and faulty disks do not exceed the redundancy of the cluster where the faulty nodes and faulty disks are located.
[0088] The method of this embodiment can send a fault test request to each node in the distributed storage system, so that each node can perform a fault test on other nodes in the distributed storage system based on the fault test request, and feed back the fault test result to the management node in a response message corresponding to the fault test request. Then, the management node can accurately determine the faulty nodes in the distributed storage system and the fault type of each faulty node based on the response message corresponding to the fault test request fed back by each node.
[0089] Based on all the above embodiments, this application also provides a complete fault handling method, such as Figure 5 As shown, the method includes:
[0090] S501, sending a fault test request to each node in the distributed storage system;
[0091] S502, determining the faulty nodes and the fault types of the faulty nodes in the distributed storage system according to the response messages corresponding to the fault test requests fed back by the nodes;
[0092] S503, when there are faulty nodes of different fault types in the distributed storage system, for each faulty node, determine whether the fault type of the faulty node is a node fault type; if the fault type of the faulty node is a node fault type, execute S504; if the fault type of the faulty node is not a node fault type but a disk fault type, execute S505;
[0093] S504, sending a restart instruction to the faulty node; the restart instruction is used to instruct the faulty node to perform a restart operation; or disconnect the service interaction with the faulty node;
[0094] S505, determining whether there is still a node failure on the faulty node; if there is still a node failure on the faulty node, executing S506; if there is no node failure on the faulty node, executing S507;
[0095] S506, performing node failure processing on the failed node to obtain a processed node, and performing disk failure processing on the processed node;
[0096] S507, sending a disk isolation instruction to the faulty node; the disk isolation instruction is used to instruct the faulty node to isolate the faulty disk.
[0097] In addition, the embodiments of the present application also involve a method for handling service pseudo-death failures. For example, two services can detect the status of each other's services. If the local service detects that the status of the opposite service is abnormal, it can report to the main service oJmgs that the opposite service has a service pseudo-death failure. Thus, the main service oJmgs can determine whether the node where the faulty service is located is super redundant, and stop the process of the faulty service if the node where the faulty service is located is within the redundant range.
[0098] In the above fault handling method, the fault type of each faulty node in the distributed storage system is determined; when there are faulty nodes of different fault types in the distributed storage system, a preset processing strategy is used to handle the faults of each faulty node; the preset processing strategy includes processing strategies for different types of faults. Since the preset processing strategy includes processing strategies for different types of faults, for the scenario where multiple server nodes in the distributed storage system have multiple types of faults at the same time, the embodiment of the present application can use the preset processing strategy to accurately handle faulty nodes of various fault types, without the need for manual intervention for fault handling, and can implement automatic node business avoidance, kicking disks and other fault handling processes, so it can improve the fault handling efficiency of the distributed storage system, thereby improving the high availability of the distributed storage system and reducing the impact of single-node failures in the cluster of the distributed storage system on the business.
[0099] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0100] Based on the same inventive concept, the embodiment of the present application also provides a fault handling device for implementing the fault handling method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more fault handling device embodiments provided below can refer to the limitations on the fault handling method above, and will not be repeated here.
[0101] In an exemplary embodiment, Figure 6 As shown, a fault handling device is provided, comprising: a determination module 31 and a fault handling module 32, wherein:
[0102] The determination module 31 is used to determine the fault type of each faulty node in the distributed storage system.
[0103] The fault processing module 32 is used to use a preset processing strategy to perform fault processing on each fault node when there are fault nodes of different fault types in the distributed storage system; the preset processing strategy includes processing strategies for different types of faults.
[0104] In one embodiment, the preset processing strategy includes a node failure processing strategy and a disk failure processing strategy, and the failure processing module 32 includes:
[0105] A node fault processing unit is used to call a node fault processing strategy to perform fault processing on each faulty node if the fault type of the faulty node is a node fault type;
[0106] The disk fault processing unit is used to call the disk fault processing strategy to perform fault processing on each faulty node if the fault type of the faulty node is a disk fault type.
[0107] In one of the embodiments, the node failure processing unit includes:
[0108] The node fault processing subunit is used to send a restart instruction to the faulty node; the restart instruction is used to instruct the faulty node to perform a restart operation; or to disconnect the business interaction with the faulty node.
[0109] In one of the embodiments, the disk failure processing unit includes:
[0110] A node fault determination subunit is used to determine whether there are other node faults on the faulty node;
[0111] A first fault processing subunit is used for, if there are other node faults on the faulty node, performing node fault processing on the faulty node to obtain a processed node, and performing disk fault processing on the processed node;
[0112] The second fault processing subunit is used to perform disk fault processing on the faulty node if there is no node fault on the faulty node.
[0113] In one embodiment, the second fault processing subunit is specifically configured to:
[0114] A disk isolation instruction is sent to the faulty node; the disk isolation instruction is used to instruct the faulty node to isolate the faulty disk.
[0115] In one embodiment, the determination module 31 includes:
[0116] A sending unit, used to send a fault test request to each node in the distributed storage system;
[0117] The fault type determination unit is used to determine the faulty nodes and the fault types of the faulty nodes in the distributed storage system according to the response messages corresponding to the fault test requests fed back by the nodes.
[0118] Each module in the above fault handling device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0119] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, or the computer device may be a server. The internal structure diagram thereof may be as follows: Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a fault handling method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.
[0120] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0121] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0122] Determine the fault type of each faulty node in the distributed storage system;
[0123] When there are faulty nodes of different fault types in the distributed storage system, a preset processing strategy is used to process the faults of each faulty node; the preset processing strategy includes processing strategies for different types of faults.
[0124] In one embodiment, the preset processing strategy includes a node failure processing strategy and a disk failure processing strategy. The preset processing strategy is used to process the failure of each failed node. When the processor executes the computer program, the following steps are also implemented:
[0125] For each faulty node, if the fault type of the faulty node is a node fault type, the node fault handling strategy is called to handle the faulty node; if the fault type of the faulty node is a disk fault type, the disk fault handling strategy is called to handle the faulty node.
[0126] In one embodiment, the node fault handling strategy is called to handle the fault of the faulty node, and the processor further implements the following steps when executing the computer program:
[0127] Send a restart instruction to the faulty node; the restart instruction is used to instruct the faulty node to perform a restart operation;
[0128] Alternatively, the business interaction with the failed node is disconnected.
[0129] In one embodiment, the disk failure handling strategy is called to handle the failure of the failed node, and the processor further implements the following steps when executing the computer program:
[0130] Determine whether there are other node failures on the failed node;
[0131] If there are still node faults on the faulty node, node fault processing is performed on the faulty node to obtain a processed node, and disk fault processing is performed on the processed node;
[0132] If there is no node failure on the failed node, disk failure processing is performed on the failed node.
[0133] In one embodiment, disk failure processing is performed on the failed node, and the processor further implements the following steps when executing the computer program:
[0134] A disk isolation instruction is sent to the faulty node; the disk isolation instruction is used to instruct the faulty node to isolate the faulty disk.
[0135] In one embodiment, the fault type of each faulty node in the distributed storage system is determined, and the processor further implements the following steps when executing the computer program:
[0136] Sending a fault test request to each node in the distributed storage system;
[0137] According to the response message corresponding to the fault test request fed back by each node, the faulty node and the fault type of each faulty node in the distributed storage system are determined.
[0138] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0139] Determine the fault type of each faulty node in the distributed storage system;
[0140] When there are faulty nodes of different fault types in the distributed storage system, a preset processing strategy is used to process the faults of each faulty node; the preset processing strategy includes processing strategies for different types of faults.
[0141] In one embodiment, the preset processing strategy includes a node failure processing strategy and a disk failure processing strategy. The preset processing strategy is used to process the failure of each failed node. When the computer program is executed by the processor, the following steps are also implemented:
[0142] For each faulty node, if the fault type of the faulty node is a node fault type, the node fault handling strategy is called to handle the faulty node; if the fault type of the faulty node is a disk fault type, the disk fault handling strategy is called to handle the faulty node.
[0143] In one embodiment, the node fault handling strategy is called to handle the fault of the faulty node, and the computer program, when executed by the processor, further implements the following steps:
[0144] Send a restart instruction to the faulty node; the restart instruction is used to instruct the faulty node to perform a restart operation;
[0145] Alternatively, the business interaction with the failed node is disconnected.
[0146] In one embodiment, the disk failure handling strategy is called to handle the failure of the failed node, and the computer program, when executed by the processor, further implements the following steps:
[0147] Determine whether there are other node failures on the failed node;
[0148] If there are still node faults on the faulty node, node fault processing is performed on the faulty node to obtain a processed node, and disk fault processing is performed on the processed node;
[0149] If there is no node failure on the failed node, disk failure processing is performed on the failed node.
[0150] In one embodiment, disk failure processing is performed on a failed node, and when the computer program is executed by a processor, the following steps are also implemented:
[0151] A disk isolation instruction is sent to the faulty node; the disk isolation instruction is used to instruct the faulty node to isolate the faulty disk.
[0152] In one embodiment, the fault type of each faulty node in the distributed storage system is determined, and when the computer program is executed by the processor, the following steps are also implemented:
[0153] Sending a fault test request to each node in the distributed storage system;
[0154] According to the response message corresponding to the fault test request fed back by each node, the faulty node and the fault type of each faulty node in the distributed storage system are determined.
[0155] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:
[0156] Determine the fault type of each faulty node in the distributed storage system;
[0157] When there are faulty nodes of different fault types in the distributed storage system, a preset processing strategy is used to process the faults of each faulty node; the preset processing strategy includes processing strategies for different types of faults.
[0158] In one embodiment, the preset processing strategy includes a node failure processing strategy and a disk failure processing strategy. The preset processing strategy is used to process the failure of each failed node. When the computer program is executed by the processor, the following steps are also implemented:
[0159] For each faulty node, if the fault type of the faulty node is a node fault type, the node fault handling strategy is called to handle the faulty node; if the fault type of the faulty node is a disk fault type, the disk fault handling strategy is called to handle the faulty node.
[0160] In one embodiment, the node fault handling strategy is called to handle the fault of the faulty node, and the computer program, when executed by the processor, further implements the following steps:
[0161] Send a restart instruction to the faulty node; the restart instruction is used to instruct the faulty node to perform a restart operation;
[0162] Alternatively, the business interaction with the failed node is disconnected.
[0163] In one embodiment, the disk failure handling strategy is called to handle the failure of the failed node, and the computer program, when executed by the processor, further implements the following steps:
[0164] Determine whether there are other node failures on the failed node;
[0165] If there are still node faults on the faulty node, node fault processing is performed on the faulty node to obtain a processed node, and disk fault processing is performed on the processed node;
[0166] If there is no node failure on the failed node, disk failure processing is performed on the failed node.
[0167] In one embodiment, disk failure processing is performed on a failed node, and when the computer program is executed by a processor, the following steps are also implemented:
[0168] A disk isolation instruction is sent to the faulty node; the disk isolation instruction is used to instruct the faulty node to isolate the faulty disk.
[0169] In one embodiment, the fault type of each faulty node in the distributed storage system is determined, and when the computer program is executed by the processor, the following steps are also implemented:
[0170] Sending a fault test request to each node in the distributed storage system;
[0171] According to the response message corresponding to the fault test request fed back by each node, the faulty node and the fault type of each faulty node in the distributed storage system are determined.
[0172] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0173] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0174] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A fault handling method, characterized in that: Applied to a distributed storage system, the method comprises: Determining the fault type of each faulty node in the distributed storage system; When there are faulty nodes of different fault types in the distributed storage system, a preset processing strategy is used to perform fault processing on each of the faulty nodes; the preset processing strategy includes processing strategies for different types of faults.
2. The method according to claim 1, characterized in that The preset processing strategy includes a node fault processing strategy and a disk fault processing strategy. The preset processing strategy is used to perform fault processing on each faulty node, including: For each of the faulty nodes, if the fault type of the faulty node is a node fault type, the node fault handling strategy is called to handle the fault of the faulty node; if the fault type of the faulty node is a disk fault type, the disk fault handling strategy is called to handle the fault of the faulty node.
3. The method according to claim 2, characterized in that The calling of the node fault handling strategy to handle the fault of the faulty node includes: Sending a restart instruction to the faulty node; the restart instruction is used to instruct the faulty node to perform a restart operation; Alternatively, the service interaction with the faulty node is disconnected.
4. The method according to claim 2, characterized in that The calling of the disk fault handling strategy to handle the fault of the faulty node includes: Determine whether there is still a node failure on the faulty node; If the node fault still exists on the faulty node, node fault processing is performed on the faulty node to obtain a processed node, and disk fault processing is performed on the processed node; If the node failure does not exist on the faulty node, disk failure processing is performed on the faulty node.
5. The method according to claim 4, characterized in that The performing disk failure processing on the failed node includes: A disk isolation instruction is sent to the faulty node; the disk isolation instruction is used to instruct the faulty node to isolate the faulty disk.
6. The method according to any one of claims 1 to 5, characterized in that Determining the fault type of each faulty node in the distributed storage system includes: Sending a fault test request to each node in the distributed storage system; According to the response message corresponding to the fault test request fed back by each of the nodes, the faulty nodes in the distributed storage system and the fault type of each of the faulty nodes are determined.
7. A fault handling device, characterized in that: Applied to a distributed storage system, the device comprises: A determination module, used to determine the fault type of each faulty node in the distributed storage system; The fault processing module is used to adopt a preset processing strategy to perform fault processing on each fault node when there are fault nodes of different fault types in the distributed storage system; the preset processing strategy includes processing strategies for different types of faults.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.