Fault processing method and device, computer equipment, storage medium and computer program product

Abstract

The invention relates to a fault processing method and device, computer equipment, a storage medium and a computer program product, and the method comprises the steps: processing a fault controller in a disk array according to fault early warning information under the condition that the fault early warning information sent by a corresponding logic unit is received; and under the condition that the fault early warning information sent by the corresponding logic unit is not received, interacting with other controllers in the disk array, and processing the fault controller in the disk array according to the interaction information. According to the method, the logic unit is arranged on the controller mainboard to sense the state of the slot position controller, and the state is synchronized to the logic units on other slot positions, so that the fault can be quickly sensed, and the fault isolation is carried out. Moreover, when the logic unit reports the fault early warning information, other controllers can quickly sense the fault through the corresponding logic unit, so that the fault detection efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a fault handling method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the rapid development of information technology, the demand for data storage and processing is increasing. As an efficient storage system, disk arrays are widely used in various types of servers and have become an indispensable and important part of servers. Disk arrays include multiple disks and multiple controllers. Each controller is responsible for managing data read and write operations and performing data redundancy processing to cope with the risk of data loss caused by a single disk failure. However, the controller itself may also fail, especially under high load or long-term operation. If the controller failure is not handled in time, it will affect the performance of the entire disk array and even cause irreversible data loss. Therefore, it is particularly important to accurately detect and efficiently isolate controller failures in disk arrays.

[0003] Currently, when troubleshooting a controller cluster, a heartbeat mechanism is usually introduced to monitor the running status of other controllers for fault detection and troubleshooting. However, the above fault handling method has the problem of inaccuracy. Summary of the invention

[0004] Based on this, it is necessary to provide a fault handling method, apparatus, computer equipment, storage medium and computer program product that can accurately handle faults in response to the above technical problems.

[0005] In a first aspect, the present application provides a fault handling method, which is applied to any controller in a disk array, and the method comprises:

[0006] When receiving the fault warning information sent by the corresponding logic unit, processing the faulty controller in the disk array according to the fault warning information;

[0007] In the case of not receiving the fault warning information sent by the corresponding logic unit, it interacts with other controllers in the disk array and processes the faulty controller in the disk array according to the interaction information.

[0008] The fault handling method provided in the embodiment of the present application processes the fault controller in the disk array according to the fault warning information when the fault warning information sent by the corresponding logic unit is received, and interacts with other controllers in the disk array when the fault warning information sent by the corresponding logic unit is not received, and processes the fault controller in the disk array according to the interactive information. In the above method, by installing a logic unit on the controller mainboard to sense the state of the controller in the slot and synchronizing it to the logic units on other slots, the fault can be quickly sensed and the fault can be isolated. When a fault occurs, if the fault warning information is reported by the logic unit, the fault is handled by the fault warning information. If the logic unit does not report the fault warning information, the fault is handled by the interactive information between the controllers. The above method provides a fault handling method in different scenarios, which can comprehensively and accurately handle the fault, thereby improving the accuracy of fault handling. Moreover, when the fault warning information is reported by the logic unit, other controllers can quickly sense the fault through the corresponding logic unit, which can improve the efficiency of fault detection.

[0009] In one embodiment, processing a faulty controller in a disk array according to the fault warning information includes:

[0010] Determine the faulty controller according to the fault warning information;

[0011] A first competition master operation is performed with other controllers in the disk array, and the faulty controller is processed according to a first competition master result of the first competition master operation.

[0012] In one embodiment, processing the faulty controller according to the first competing master result of the first competing master operation includes:

[0013] If the first competition result determines that the controller is the master controller, the faulty controller is isolated and a notification message is sent to the non-master controller;

[0014] If the first competition result determines that the controller is not the master controller, a notification message sent by the master controller determined by the first competition result is received; the notification message is used to instruct the non-master controller to stop interacting with the faulty controller.

[0015] The method described in the embodiment of the present application can effectively prevent the faulty controller from affecting the entire disk array by timely identifying and isolating the faulty controller, thereby improving the overall stability and reliability of the system. Moreover, when a fault occurs, the main controller can be quickly identified by using the competitive master operation, which helps the system to quickly resume normal operation, shorten the fault recovery time as much as possible, and improve the availability of the system.

[0016] In one embodiment, interacting with other controllers in the disk array and processing a faulty controller in the disk array according to the interaction information includes:

[0017] Interact with other controllers in the disk array, and determine the faulty controller and the main controller in the disk array according to the fault identifier carried in the interactive information;

[0018] A fault handling instruction is sent to the logic unit corresponding to the main controller; the fault handling instruction is used to instruct the logic unit corresponding to the main controller to handle the faulty controller in the disk array.

[0019] In one embodiment, interacting with other controllers in the disk array and determining the faulty controller and the main controller in the disk array according to the fault identifier carried in the interaction information includes:

[0020] If the interactive information does not carry a fault identifier, a second master competition operation is performed with other controllers to determine the faulty controller and the master controller;

[0021] If the interaction information carries a fault identifier, the faulty controller is determined according to the fault identifier, and a third master competition operation is performed with other controllers to determine the master controller.

[0022] The method described in the embodiment of the present application determines the fault type through the content of the interactive information, and executes corresponding fault handling strategies for different fault types, so as to accurately isolate the fault. Moreover, through the transmission of logical units between each slot and the execution of the action of forcibly killing the faulty controller, it can ensure that the faulty controller is quickly isolated, thereby improving the security of the storage system.

[0023] In one embodiment, performing a second master-competing operation with other controllers to determine a faulty controller and a master controller includes:

[0024] Determine connectivity with other controllers;

[0025] Determining the overall connectivity of the controller based on connectivity with other controllers;

[0026] A second competing master result of the second competing master operation is determined based on the comprehensive connectivity and the comprehensive connectivity of other controllers; the second competing master result includes determining the controller with the lowest comprehensive connectivity as the faulty controller, and determining the controller with the highest comprehensive connectivity as the master controller.

[0027] The method described in the embodiments of the present application can quickly identify the controller with the problem by determining that the controller with the lowest comprehensive connectivity is the faulty controller when a communication failure occurs in the disk array, and can ensure that the system is handled by the most stable and reliable controller by determining that the controller with the highest comprehensive connectivity is the main controller, thereby improving the stability and reliability of the system.

[0028] In a second aspect, the present application further provides a fault handling method, which is applied to any logical unit in a disk array, and the method comprises:

[0029] When a corresponding controller is detected to have a fault, a fault warning message is sent to other logical units; the fault warning message is used to instruct other controllers corresponding to other logical units to process the faulty controller in the disk array according to the fault warning message;

[0030] When receiving the fault processing instruction sent by the corresponding controller, the faulty controller in the disk array is processed.

[0031] In one embodiment, the method further comprises:

[0032] When receiving fault warning information sent by other logic units, the fault warning information is sent to the corresponding controller.

[0033] In one embodiment, processing a failed controller in a disk array includes:

[0034] Determine the faulty controller in the disk array according to the fault handling instruction and generate a reset instruction;

[0035] Sending a reset instruction to the target logic unit corresponding to the faulty controller; the reset instruction is used to instruct the target logic unit to perform a reset operation on the faulty controller;

[0036] A reset success message returned by the target logic unit is received, and the reset success message is sent to the corresponding controller; the reset success message is used to instruct the corresponding controller to isolate the faulty controller.

[0037] The fault handling method provided in the embodiment of the present application sends fault warning information to other logic units when a fault occurs in the corresponding controller, and processes the faulty controller in the disk array when a fault handling instruction sent by the corresponding controller is received. Among them, the fault warning information is used to instruct other controllers corresponding to other logic units to process the faulty controller in the disk array according to the fault warning information. In the above method, by installing a logic unit on the controller mainboard to sense the state of the controller in this slot, and synchronizing it to the logic units on other slots, the fault can be quickly sensed and the fault can be isolated. When a fault occurs, if the fault warning information is reported by the logic unit, the fault is handled through the fault warning information. If the logic unit does not report the fault warning information, the fault is handled by the interactive information between the controllers. The above method provides a fault handling method under different scenarios, which can comprehensively and accurately handle the fault, thereby improving the accuracy of fault handling. Moreover, when the fault warning information is reported by the logic unit, other controllers can quickly sense the fault through the corresponding logic unit, which can improve the efficiency of fault detection.

[0038] In a third aspect, the present application provides a fault handling device, the device comprising:

[0039] A first processing module is used to process a faulty controller in the disk array according to the fault warning information when receiving the fault warning information sent by the corresponding logic unit;

[0040] The second processing module is used to interact with other controllers in the disk array and process the faulty controller in the disk array according to the interaction information when the fault warning information sent by the corresponding logic unit is not received.

[0041] In a fourth aspect, the present application further provides a fault handling device, the device comprising:

[0042] The sending module is used to send fault warning information to other logical units when a corresponding controller fails. The fault warning information is used to instruct other controllers corresponding to other logical units to process the faulty controller in the disk array according to the fault warning information.

[0043] The third processing module is used to process the faulty controller in the disk array when receiving the fault processing instruction sent by the corresponding controller.

[0044] In a fifth aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the fault handling method described in any one of the embodiments of the first and second aspects above is implemented.

[0045] In a sixth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the fault handling method described in any embodiment of the first aspect and the second aspect above is implemented.

[0046] In a seventh aspect, the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the fault handling method described in any embodiment of the first aspect and the second aspect.

[0047] The above fault handling method, device, computer equipment, storage medium and computer program product, the method processes the fault controller in the disk array according to the fault warning information when the fault warning information sent by the corresponding logic unit is received, and interacts with other controllers in the disk array when the fault warning information sent by the corresponding logic unit is not received, and processes the fault controller in the disk array according to the interactive information. In the above method, by installing a logic unit on the controller mainboard to sense the state of the controller in the slot and synchronizing it to the logic units on other slots, the fault can be quickly sensed and the fault can be isolated. When a fault occurs, if the fault warning information is reported by the logic unit, the fault is handled by the fault warning information. If the logic unit does not report the fault warning information, the fault is handled by the interactive information between the controllers. The above method provides a fault handling method in different scenarios, which can comprehensively and accurately handle the fault, thereby improving the accuracy of fault handling. Moreover, when the fault warning information is reported by the logic unit, other controllers can quickly sense the fault through the corresponding logic unit, which can improve the efficiency of fault detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the structure of a disk array in one embodiment;

[0049] Figure 2 A schematic diagram of a flow chart of a fault handling method in an embodiment;

[0050] Figure 3 is a flowchart of a fault handling method in another embodiment;

[0051] Figure 4 is a flowchart of a fault handling method in another embodiment;

[0052] Figure 5 is a flowchart of a method for handling hardware failures in another embodiment;

[0053] Figure 6 is a flowchart of a fault handling method in another embodiment;

[0054] Figure 7 is a flowchart of a fault handling method in another embodiment;

[0055] Figure 8 A flow chart of a communication failure processing method in one embodiment;

[0056] Fig. 9 A schematic diagram of a flow chart of a method for handling a suspended animation fault in an embodiment;

[0057] Fig.10 is a flowchart of a fault handling method in another embodiment;

[0058] Fig.11 is a flowchart of a fault handling method in another embodiment;

[0059] Fig.12 is a flowchart of a fault handling method in another embodiment;

[0060] Fig.13 is a structural block diagram of a fault handling device in an embodiment;

[0061] Fig.14 is a structural block diagram of a fault handling device in an embodiment;

[0062] Fig.15 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] With the rapid development of information technology, the demand for data storage and processing is increasing. As an efficient storage system, disk arrays are widely used in various types of servers and have become an indispensable and important part of servers. The disk array includes multiple disks and multiple controllers. Each controller is responsible for managing the read and write operations of data and performing data redundancy processing to cope with the risk of data loss that may be caused by a single disk failure. However, the controller itself may also fail, especially under high load or long-term operation. If the controller failure is not handled in time, it will affect the performance of the entire disk array and even cause irreversible data loss. Therefore, it is particularly important to accurately detect and efficiently isolate controller failures in disk arrays. At present, when a controller cluster is fault-handled, a heartbeat mechanism is usually introduced to monitor the operating status of other controllers for fault detection and fault handling. However, the above fault handling method has the problem of inaccuracy. The present application provides a fault handling method to solve the above technical problems. The following embodiments will specifically illustrate the fault handling method described in the present application.

[0065] The fault handling method provided in the embodiment of the present application can be applied to Figure 1 In the disk array shown, the disk array includes a plurality of slots 10, each of which is provided with a controller 101 and a logic unit 102. The controller 101 and the logic unit 102 on the same slot 10 are connected, the controllers 101 on different slots 10 are connected to each other, and the logic units 102 on different slots 10 are connected to each other. The logic unit 102 may be a complex programmable logic device (CPLD for short). The controller 101 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, portable wearable devices, etc. The controller 101 may be implemented using an independent server or a server cluster consisting of multiple servers.

[0066] Those skilled in the art will understand that Figure 1 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the storage system to which the solution of the present application is applied. The specific storage system may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0067] In one embodiment, Figure 2 As shown, a fault handling method is provided, which is applied to Figure 1 Take any controller in the disk array in the example as an example, including the following steps:

[0068] S201 , when receiving fault warning information sent by a corresponding logic unit, processing a faulty controller in a disk array according to the fault warning information.

[0069] The fault warning information includes an identifier of a faulty controller where a hardware fault occurs. The identifier of the faulty controller may be at least one of a faulty controller ID, a faulty controller number, a faulty controller location, and the like.

[0070] In an embodiment of the present application, during the operation of the disk array, the logic unit on each slot detects the hardware status of the controller on the corresponding slot in real time. For any logic unit, when a hardware failure is detected in the controller on the slot, such as poor power contact, battery failure, hard disk failure, etc., a fault warning message can be generated according to the hardware failure, and the fault warning message can be sent to the logic units on other slots. After the logic units on other slots receive the fault warning message, the fault warning message can be sent to the controller on the corresponding slot, and then the controllers on other slots can receive the fault warning message, and after the controllers on other slots receive the fault warning message, the fault controller in the disk array can be processed according to the fault warning message, and the logical services carried by the fault controller can be taken over.

[0071] S202: When no fault warning information sent by the corresponding logic unit is received, interact with other controllers in the disk array, and process the faulty controller in the disk array according to the interaction information.

[0072] The interaction information includes communication information and working status information of each controller.

[0073] In the embodiment of the present application, if the disk array fails, and the controller does not receive the fault warning information sent by the logic unit on the slot, then it can be determined that the fault type is not a hardware fault type. The fault type can be further determined based on the interactive information of each controller in the disk array, and the fault type also includes a communication fault type and a working state fault type, wherein the communication fault type indicates that each controller has a controller that cannot communicate normally during the communication process, and the working state fault type indicates that each controller has a controller with a pseudo-death fault during the work process, that is, there is a controller with read-only, slow disk or low working efficiency. For the above two types of faults, two fault handling methods are respectively corresponding. The first is that during the operation of each controller in the disk array, each controller will send heartbeat messages to each other according to the heartbeat mechanism, and determine whether a communication fault occurs according to the reply of the heartbeat message. If a controller does not receive a reply message from the other controller within a preset time, then the controller and the other controller may both fail. In this case, the interactive information does not include the fault controller identifier, so it is necessary to further determine the fault controller according to the interaction between each controller and other controllers, and then perform fault handling on the fault controller. The second is that the controllers in the disk array will monitor each other's working status in real time during operation, and determine whether there is a controller that has been suspended based on the monitored working status information. In this scenario, the interaction information includes the faulty controller identifier, so the faulty controller can be processed based on the faulty controller identifier.

[0074] The fault handling method provided in the embodiment of the present application processes the fault controller in the disk array according to the fault warning information when the fault warning information sent by the corresponding logic unit is received, and interacts with other controllers in the disk array when the fault warning information sent by the corresponding logic unit is not received, and processes the fault controller in the disk array according to the interactive information. In the above method, by installing a logic unit on the controller mainboard to sense the state of the controller in the slot and synchronizing it to the logic units on other slots, the fault can be quickly sensed and the fault can be isolated. When a fault occurs, if the fault warning information is reported by the logic unit, the fault is handled by the fault warning information. If the logic unit does not report the fault warning information, the fault is handled by the interactive information between the controllers. The above method provides a fault handling method in different scenarios, which can comprehensively and accurately handle the fault, thereby improving the accuracy of fault handling. Moreover, when the fault warning information is reported by the logic unit, other controllers can quickly sense the fault through the corresponding logic unit, which can improve the efficiency of fault detection.

[0075] In one embodiment, a fault handling method is also provided when a fault warning message sent by a corresponding logic unit is received, such as Figure 3As shown, the "processing the faulty controller in the disk array according to the fault warning information" in the above step S201 includes:

[0076] S301, determining a faulty controller according to fault warning information.

[0077] In the embodiment of the present application, after receiving the fault warning information sent by the logic unit on the slot, the controller can determine the fault controller identifier from the fault warning information, and then determine the fault controller according to the fault controller identifier. Similarly, other controllers in the disk array can also determine the fault controller in this way.

[0078] S302: Perform a first master competition operation with other controllers in the disk array, and process the faulty controller according to a first master competition result of the first master competition operation.

[0079] The first master competition operation is to elect a master controller from multiple controllers through an election rule. The first master competition result includes a master controller and a slave controller.

[0080] In the embodiment of the present application, after determining the faulty controller, the controller and other normal controllers in the disk array can perform the first master competition operation through a preset election rule or a preset election algorithm, and then obtain the first master competition result.

[0081] Specifically, the first competition result includes two situations: the first one is that the above controller is the master controller and the other controllers are slave controllers; the second one is that one of the other controllers is the master controller and the above controller is the slave controller. Figure 4 As shown, the "processing the faulty controller according to the first competing master result of the first competing master operation" in the above step S302 specifically includes:

[0082] S3021, determine whether the controller is the main controller, if so, execute S3022, if not, execute S3023.

[0083] In the embodiment of the present application, after obtaining the first competition result, the controller can determine whether it is the main controller.

[0084] S3022, isolate the faulty controller and send notification information to the non-master controller.

[0085] The notification information is used to instruct the non-master controller to stop interacting with the faulty controller.

[0086] In an embodiment of the present application, if the first competition for the master determines that the controller is the master controller, the faulty controller is isolated, and a notification message is sent to the non-master controller, and the logical tasks carried by the faulty processor are taken over. The specific takeover method can be that the master controller bears all the tasks, or the logical tasks carried by the faulty processor can be allocated to the slave controller, or to an idle controller.

[0087] S3023, receiving notification information sent by the main controller whose first competition result is determined.

[0088] In the embodiment of the present application, if the first competition result determines that the controller is not the master controller, the notification information sent by the master controller determined by the first competition result is received, and when the master controller allocates the logical tasks carried by the faulty processor, the allocated logical tasks are processed. The corresponding fault handling method when a hardware fault occurs in the disk array can be found in Figure 5 As shown in the figure, controller 1 is used as the faulty controller and controller 2 is used as the main controller. Controller 1 on slot 1 starts to carry some logical services at time T1. At T2, a controller failure occurs and the service can no longer be provided. CPLD1 on slot 1 quickly identifies the failure and notifies CPLD2 on slot 2. At T4, CPLD2 completes the failure identification and reports to controller 2. Controller 2 isolates controller 1 from the array at time T5 and takes over the logical services on controller 1 at time T6 to continue to provide services. Since the controller failure is a deterministic hardware failure, it can be sensed by controller 2 very promptly, and after sensing, the fault can be directly isolated without further processing. The delay from T1 to T6 can be controlled at the millisecond level.

[0089] The method described in the embodiment of the present application can effectively prevent the faulty controller from affecting the entire disk array by timely identifying and isolating the faulty controller, thereby improving the overall stability and reliability of the system. Moreover, when a fault occurs, the main controller can be quickly identified by using the competitive master operation, which helps the system to quickly resume normal operation, shorten the fault recovery time as much as possible, and improve the availability of the system.

[0090] In one embodiment, the fault handling method when the fault warning information sent by the corresponding logic unit is not received is as follows: Figure 6 As shown, the above step S202 of "interacting with other controllers in the disk array and processing the faulty controller in the disk array according to the interaction information" includes:

[0091] S401, interacting with other controllers in the disk array, and determining a faulty controller and a main controller in the disk array according to a fault identifier carried in the interaction information.

[0092] The interaction information carries a fault identifier. Optionally, the interaction information does not carry a fault identifier.

[0093] In the embodiment of the present application, if a disk array fails and the controller does not receive the fault warning information sent by the logic unit on the slot, it can be determined that the fault type is not a hardware fault type. The controller can interact with other controllers in the disk array to obtain interaction information, and then further determine the faulty controller and the main controller in the disk array based on whether the interaction information carries a fault identifier.

[0094] Specifically, Figure 7 As shown, the above step S401 of "interacting with other controllers in the disk array and determining the faulty controller and the main controller in the disk array according to the fault identifier carried in the interaction information" includes:

[0095] S4011, determine whether the interaction information carries a fault identifier. If the interaction information does not carry a fault identifier, execute S4012; if the interaction information carries a fault identifier, execute S4013.

[0096] In the embodiment of the present application, after acquiring the interaction information, the controller may further determine whether the interaction information carries a fault identifier.

[0097] S4012, perform a second master competition operation with other controllers to determine the faulty controller and the master controller.

[0098] In an embodiment of the present application, if the interactive information does not carry a fault identifier, this situation indicates that a communication failure has occurred, that is, a brain split between controllers. For a brain split scenario, the controllers are alive but cannot communicate with each other. However, the disk array provides highly reliable services to the outside world, so the received data requests will be backed up in the array to improve redundancy. If the network between the controllers fails, mirror synchronization of data cannot be performed, and the controller that cannot be connected needs to be fault-isolated from the array. Therefore, when it is determined that a communication failure has occurred, it is necessary to further determine the faulty controller, and specifically, a second competitive master operation can be performed with other controllers to determine the faulty controller and the main controller.

[0099] S4013, determine the faulty controller according to the fault identifier, and perform a third master competition operation with other controllers to determine the master controller.

[0100] In the embodiment of the present application, if the interactive information carries a fault identifier, this situation indicates that a pseudo-death fault has occurred. If the controller pseudo-death phenomenon occurs, the tasks of the controller cannot be executed normally, which will cause abnormal heartbeat monitoring of the normal controller. Therefore, when a pseudo-death fault occurs, the faulty controller can be determined according to the fault identifier, and then the third master competition operation is performed with other controllers, and the controller that successfully competes for the master is determined as the master controller, and the controller that fails to compete for the master is determined as the slave controller.

[0101] S402: Send a fault handling instruction to a logic unit corresponding to the main controller.

[0102] The fault handling instruction is used to instruct the logic unit corresponding to the main controller to handle the faulty controller in the disk array. The fault handling instruction may specifically reset the faulty controller, then isolate the faulty controller, and take over the logical services running on the faulty controller.

[0103] In an embodiment of the present application, after the controller determines the faulty controller and the main controller based on the above steps, the main controller can send a fault handling instruction to the logic unit on the current slot. After the logic unit corresponding to the main controller receives the fault handling instruction, it can send the fault handling instruction to the logic unit corresponding to the faulty controller. When the logic unit receives the fault handling instruction sent by the corresponding controller, it sends the fault instruction to other logic units. After other logic units receive the fault handling instruction, they can identify the faulty controller in the disk array and generate a reset instruction, and then send the reset instruction to the target logic unit corresponding to the faulty controller. After the target logic unit receives the reset instruction, it can reset the CPU of the faulty controller to reset the faulty controller, and return a reset success message to the logic unit. After the logic unit receives the reset success message, it sends the reset success message to the corresponding controller. After the controller receives the reset success message, it notifies other controllers to stop interacting with the faulty controller, isolates the faulty controller, and takes over the logical services running on the faulty controller.

[0104] For example, the corresponding troubleshooting methods when a disk array communication failure occurs can be found in Figure 8As shown, controller 1 is used as the main controller and controller 2 is used as the faulty controller for explanation. Specifically, the split-brain scenario of two controllers is handled as an example. If it is a multi-controller scenario, it is also handled in pairs. After the split-brain, controller 1 and controller 2 sense it separately and compete for the master. The master competition algorithm can adopt a processing similar to the RAFT protocol, and the connectivity of each controller with other controllers is added to the competition for the master, so that the controller with high connectivity competes for the master. The party that fails in the competition for the master will no longer perform subsequent processing. After the successful competition for the master kills the opposite controller, it will isolate the opposite controller from the array and take over the logical services carried by the opposite controller. For example, the corresponding fault handling method when a false death fault occurs in the disk array can be seen in Fig. 9 As shown in the figure, controller 1 is the main controller and controller 2 is the faulty controller. If the controller is suspended, the tasks of the controller cannot be executed normally, which will cause abnormal heartbeat monitoring of the normal controller and trigger split-brain perception, that is, jump to the fault handling mode of communication failure.

[0105] The method described in the embodiment of the present application determines the fault type through the content of the interactive information, and executes corresponding fault handling strategies for different fault types, so as to accurately isolate the fault. Moreover, through the transmission of logical units between each slot and the execution of the action of forcibly killing the faulty controller, it can ensure that the faulty controller is quickly isolated, thereby improving the security of the storage system.

[0106] In one embodiment, a specific implementation method of performing a third master competition operation with other controllers to determine a faulty controller and a master controller is also provided, such as Fig.10 As shown, the "perform a second master competition operation with other controllers to determine the faulty controller and the master controller" in the above step S4013 includes:

[0107] S501, determining connectivity with other controllers.

[0108] Among them, connectivity is used to indicate the interaction of heartbeat messages between controllers.

[0109] In the embodiment of the present application, when it is determined that a communication failure has occurred, each controller may determine the connectivity with other controllers based on the interaction of heartbeat messages.

[0110] S502: Determine the comprehensive connectivity of the controller according to the connectivity with other controllers.

[0111] Among them, comprehensive connectivity is used to represent the interaction between the controller and all other controllers.

[0112] In the embodiment of the present application, for any controller, after the connectivity with each other controller is obtained based on the above steps, the comprehensive connectivity of the controller can be further determined according to the connectivity with each controller.

[0113] S503: Determine a second host competing result of the second host competing operation according to the comprehensive connectivity and the comprehensive connectivity of other controllers.

[0114] The second competition result includes determining the controller with the lowest comprehensive connectivity as the faulty controller, and determining the controller with the highest comprehensive connectivity as the master controller.

[0115] In an embodiment of the present application, each controller obtains its own comprehensive connectivity and the comprehensive connectivity of other controllers based on the above steps. According to the combined connectivity and the comprehensive connectivity of other controllers, the controller with the lowest comprehensive connectivity can be regarded as the faulty controller, and the controller with the highest comprehensive connectivity can be regarded as the main controller, thereby obtaining the second main competition result.

[0116] The method described in the embodiments of the present application can quickly identify the controller with the problem by determining that the controller with the lowest comprehensive connectivity is the faulty controller when a communication failure occurs in the disk array, and can ensure that the system is handled by the most stable and reliable controller by determining that the controller with the highest comprehensive connectivity is the main controller, thereby improving the stability and reliability of the system.

[0117] In one embodiment, a fault handling method is also provided. Figure 1 Take any logical unit in the disk array as an example. Fig.11 As shown, the following steps are included:

[0118] S601, when a corresponding controller is detected to have a fault, fault warning information is sent to other logical units; the fault warning information is used to instruct other controllers corresponding to other logical units to process the faulty controller in the disk array according to the fault warning information.

[0119] S602: When receiving a fault processing instruction sent by a corresponding controller, process the faulty controller in the disk array.

[0120] S603, when receiving the fault warning information sent by other logic units, send the fault warning information to the corresponding controller.

[0121] Specifically, Fig.12 As shown, the "processing the faulty controller in the disk array" in the above step S602 includes:

[0122] S6021: Determine a faulty controller in the disk array according to the fault handling instruction, and generate a reset instruction.

[0123] S6022, sending a reset instruction to a target logic unit corresponding to the faulty controller; the reset instruction is used to instruct the target logic unit to perform a reset operation on the faulty controller.

[0124] S6022, receiving a reset success message returned by the target logic unit, and sending the reset success message to the corresponding controller; the reset success message is used to instruct the corresponding controller to isolate the faulty controller.

[0125] The methods described in the above steps are all described in the above embodiments. Please refer to the above description for details and will not be repeated here.

[0126] The fault handling method provided in the embodiment of the present application sends fault warning information to other logic units when a fault occurs in the corresponding controller, and processes the faulty controller in the disk array when a fault handling instruction sent by the corresponding controller is received. Among them, the fault warning information is used to instruct other controllers corresponding to other logic units to process the faulty controller in the disk array according to the fault warning information. In the above method, by installing a logic unit on the controller mainboard to sense the state of the controller in this slot, and synchronizing it to the logic units on other slots, the fault can be quickly sensed and the fault can be isolated. When a fault occurs, if the fault warning information is reported by the logic unit, the fault is handled through the fault warning information. If the logic unit does not report the fault warning information, the fault is handled by the interactive information between the controllers. The above method provides a fault handling method under different scenarios, which can comprehensively and accurately handle the fault, thereby improving the accuracy of fault handling. Moreover, when the fault warning information is reported by the logic unit, other controllers can quickly sense the fault through the corresponding logic unit, which can improve the efficiency of fault detection.

[0127] Based on all the above embodiments, a fault handling method is also provided, which includes:

[0128] When the logic unit detects that the corresponding controller has failed, that is, the disk array has a hardware failure, it sends fault warning information to other logic units. When other logic units receive the fault warning information sent by the logic unit corresponding to the faulty controller, they send the fault warning information to the corresponding controller, and the controller can receive the fault warning information sent by the corresponding logic unit. When the controller receives the fault warning information sent by the corresponding logic unit, it determines the faulty controller according to the fault warning information and performs the first competition operation with other controllers in the disk array. When the first competition result determines that the controller is the master controller, the faulty controller is isolated and notification information is sent to the non-master controller; when the first competition result determines that the controller is not the master controller, the notification information sent by the master controller determined by the first competition result is received. The notification information is used to instruct the non-master controller to stop interacting with the faulty controller.

[0129] When the controller does not receive the fault warning information sent by the corresponding logic unit, it interacts with other controllers in the disk array and determines whether the interactive information carries a fault identifier. There are two situations: when the interactive information does not carry a fault identifier, that is, a communication failure occurs in the disk array, the controller can determine the connectivity with other controllers, and then determine the comprehensive connectivity of the controller based on the connectivity with other controllers, and finally determine the second competition result of the second competition operation based on the comprehensive connectivity and the comprehensive connectivity of other controllers. Among them, the second competition result includes determining that the controller with the lowest comprehensive connectivity is the fault controller, and determining that the controller with the highest comprehensive connectivity is the master controller. When the interactive information carries a fault identifier, that is, a pseudo-death failure occurs in the disk array, the controller can determine the fault controller based on the fault identifier, and perform a third competition operation with other controllers to determine the master controller, and then send a fault processing instruction to the logic unit corresponding to the master controller. Among them, the fault processing instruction is used to instruct the logic unit corresponding to the master controller to process the fault controller in the disk array.

[0130] When the logical unit receives the fault handling instruction sent by the corresponding controller, it sends the fault instruction to other logical units. After receiving the fault handling instruction, the other logical units can find the faulty controller in the disk array and generate a reset instruction, and then send the reset instruction to the target logical unit corresponding to the faulty controller. After receiving the reset instruction, the target logical unit can reset the faulty controller and return a reset success message to the logical unit. After receiving the reset success message, the logical unit sends the reset success message to the corresponding controller. After receiving the reset success message, the controller notifies other controllers to stop interacting with the faulty controller, isolates the faulty controller, and takes over the logical services running on the faulty controller.

[0131] The methods described in the above steps are all described in the above embodiments. Please refer to the above description for details and will not be repeated here.

[0132] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0133] Based on the same inventive concept, the embodiment of the present application also provides a fault handling device for implementing the fault handling method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more fault handling device embodiments provided below can refer to the limitations on the fault handling method above, and will not be repeated here.

[0134] In one embodiment, Fig.13 As shown, a fault handling device is provided, comprising:

[0135] The first processing module 11 is used to process the faulty controller in the disk array according to the faulty warning information when receiving the faulty warning information sent by the corresponding logic unit.

[0136] The second processing module 12 is used to interact with other controllers in the disk array and process the faulty controller in the disk array according to the interaction information when the fault warning information sent by the corresponding logic unit is not received.

[0137] In one embodiment, the first processing module 11 includes:

[0138] The first determining unit is used to determine the faulty controller according to the fault warning information.

[0139] The first competing master unit is used to perform a first competing master operation with other controllers in the disk array and process the faulty controller according to the first competing master result of the first competing master operation.

[0140] In one embodiment, the first processing unit includes:

[0141] The processing subunit is used for isolating the faulty controller and sending notification information to the non-master controller if the first master competition result determines that the controller is the master controller.

[0142] The receiving subunit is used to receive notification information sent by the main controller determined by the first competition for the main controller result if the controller is not the main controller; the notification information is used to instruct the non-main controller to stop interacting with the faulty controller.

[0143] In one embodiment, the second processing module 12 includes:

[0144] The second determining unit is used to interact with other controllers in the disk array and determine the faulty controller and the main controller in the disk array according to the fault identifier carried in the interaction information.

[0145] The first sending unit is used to send a fault processing instruction to the logic unit corresponding to the main controller; the fault processing instruction is used to instruct the logic unit corresponding to the main controller to process the faulty controller in the disk array.

[0146] In one embodiment, the second determining unit includes:

[0147] The second competing master subunit is used to perform a second competing master operation with other controllers to determine the faulty controller and the main controller if the interactive information does not carry a fault identifier.

[0148] The third competing master sub-unit is used to determine the faulty controller according to the fault identifier if the interactive information carries the fault identifier, and perform the third competing master operation with other controllers to determine the master controller.

[0149] In one embodiment, the above-mentioned second competing master subunit is specifically used to determine the connectivity with other controllers; determine the comprehensive connectivity of the controller based on the connectivity with other controllers; determine the second competing master result of the second competing master operation based on the comprehensive connectivity and the comprehensive connectivity of other controllers; the second competing master result includes determining that the controller with the lowest comprehensive connectivity is the faulty controller, and determining the controller with the highest comprehensive connectivity is the master controller.

[0150] In one embodiment, Fig.14 As shown, a fault handling device is also provided, comprising:

[0151] The sending module 13 is used to send fault warning information to other logical units when a corresponding controller fails. The fault warning information is used to instruct other controllers corresponding to other logical units to process the faulty controller in the disk array according to the fault warning information.

[0152] The third processing module 14 is used to process the faulty controller in the disk array when receiving a fault processing instruction sent by the corresponding controller.

[0153] In one embodiment, the fault handling device further includes:

[0154] The sending module is used to send the fault warning information to the corresponding controller when receiving the fault warning information sent by other logic units.

[0155] In one embodiment, the third processing module 14 includes:

[0156] The generating unit is used to determine the faulty controller in the disk array according to the fault handling instruction and generate a reset instruction.

[0157] The second sending unit is used to send a reset instruction to a target logic unit corresponding to the fault controller; the reset instruction is used to instruct the target logic unit to perform a reset operation on the fault controller.

[0158] The receiving unit is used to receive a reset success message returned by the target logic unit and send the reset success message to the corresponding controller; the reset success message is used to instruct the corresponding controller to isolate the faulty controller.

[0159] Each module in the above fault handling device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0160] In one embodiment, a computer device is provided. The computer device may be a terminal or a server. The internal structure diagram thereof may be as follows: Fig.15As shown, the computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a fault handling method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0161] Those skilled in the art will understand that Fig.15 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0162] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the method described in any of the above embodiments are implemented.

[0163] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0164] In one embodiment, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0165] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0166] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A fault handling method, characterized in that: Applied to any controller in a disk array, the method comprises: In case of receiving the fault warning information sent by the corresponding logic unit, processing the faulty controller in the disk array according to the fault warning information; In the case of not receiving the fault warning information sent by the corresponding logic unit, interacting with other controllers in the disk array, and processing the faulty controller in the disk array according to the interactive information.

2. The method according to claim 1, characterized in that The processing of the faulty controller in the disk array according to the fault warning information includes: Determining the fault controller according to the fault warning information; A first competing master operation is performed with other controllers in the disk array, and the faulty controller is processed according to a first competing master result of the first competing master operation.

3. The method according to claim 2, characterized in that The processing of the faulty controller according to the first competing master result of the first competing master operation includes: If the first competition result determines that the controller is the master controller, isolating the faulty controller and sending notification information to the non-master controller; If the first competition result determines that the controller is not the master controller, a notification message sent by the master controller determined by the first competition result is received; the notification message is used to instruct the non-master controller to stop interacting with the faulty controller.

4. The method according to any one of claims 1 to 3, characterized in that: The interacting with other controllers in the disk array and processing a faulty controller in the disk array according to the interaction information includes: Interacting with other controllers in the disk array, and determining the faulty controller and the main controller in the disk array according to the fault identifier carried in the interaction information; A fault processing instruction is sent to the logic unit corresponding to the main controller; the fault processing instruction is used to instruct the logic unit corresponding to the main controller to process the faulty controller in the disk array.

5. The method according to claim 4, characterized in that The step of determining the faulty controller and the main controller in the disk array according to the fault identifier carried in the interactive information includes: If the interaction information does not carry the fault identifier, performing a second master competition operation with the other controllers to determine the faulty controller and the master controller; If the interaction information carries the fault identifier, the faulty controller is determined according to the fault identifier, and a third master competition operation is performed with the other controllers to determine the master controller.

6. The method according to claim 5, characterized in that The performing a second master competition operation with the other controllers to determine the faulty controller and the master controller includes: Determining connectivity with the other controllers; Determining the comprehensive connectivity of the controller according to the connectivity with the other controllers; Based on the comprehensive connectivity and the comprehensive connectivity of the other controllers, a second competing master result of the second competing master operation is determined; the second competing master result includes determining the controller with the lowest comprehensive connectivity as the faulty controller, and determining the controller with the highest comprehensive connectivity as the main controller.

7. A fault handling method, characterized in that: Applied to any logical unit in a disk array, the method comprises: When a corresponding controller is detected to have failed, a fault warning message is sent to other logical units; the fault warning message is used to instruct other controllers corresponding to the other logical units to process the failed controller in the disk array according to the fault warning message; When a fault processing instruction sent by a corresponding controller is received, the faulty controller in the disk array is processed.

8. The method according to claim 7, characterized in that The method further comprises: When the fault warning information sent by the other logic units is received, the fault warning information is sent to the corresponding controller.

9. The method according to claim 7 or 8, characterized in that: The processing of the faulty controller in the disk array includes: Determine a faulty controller in the disk array according to the fault handling instruction, and generate a reset instruction; Sending the reset instruction to the target logic unit corresponding to the fault controller; the reset instruction is used to instruct the target logic unit to perform a reset operation on the fault controller; A reset success message returned by the target logic unit is received, and the reset success message is sent to a corresponding controller; the reset success message is used to instruct the corresponding controller to isolate the faulty controller.

10. A fault handling device, characterized in that: The device comprises: A first processing module, configured to process a faulty controller in the disk array according to the fault warning information when receiving the fault warning information sent by the corresponding logic unit; The second processing module is used to interact with other controllers in the disk array and process the faulty controller in the disk array according to the interaction information when the fault warning information sent by the corresponding logic unit is not received.

Images