Storage system fault handling method, device, product, storage system and medium

Abstract

The present invention discloses a storage system fault handling method, device, product, storage system and medium, and relates to the field of storage technology. According to the distribution of the target hard disk (i.e., the hard disk with a single port detected by the control node) and the target chassis (i.e., the chassis with a single controller detected by the control node) in the current link, the fault module is determined, and the fault repair strategy is used to repair the fault module; and there are at least two control nodes in the main cabinet, each chassis connected to the main cabinet includes at least two controllers, at least one controller in each chassis is connected to the first control node, and at least another controller is connected to the second control node, and each port of each hard disk is connected to a corresponding controller, that is, the controller node redundancy setting, the controller redundancy setting and the hard disk port redundancy setting ensure the redundancy and reliability of the storage system; the link formed by connecting all chassis with the same port of the main cabinet is used as the detection unit, which improves the efficiency of fault handling.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a storage system fault handling method, device, product, storage system and medium. Background Art

[0002] With the rapid development of information technology and the rapid growth of data volume, storage systems have become an indispensable part of the computing environment. The reliability, availability, and serviceability (RAS) of storage systems directly affect the normal operation of services.

[0003] However, traditional storage systems face a variety of potential failure risks. For example, hard disks, controllers, and chassis may all fail, resulting in data loss or service interruption, affecting user business.

[0004] It can be seen that how to ensure the reliability, availability and serviceability of the storage system is a technical problem that people in this field need to solve urgently. Summary of the invention

[0005] The purpose of the present invention is to provide a storage system fault handling method, device, product, storage system and medium to solve the technical problem that data loss or service interruption in the storage system affects user business.

[0006] In order to solve the above technical problems, the present invention provides a storage system fault handling method, the method comprising:

[0007] Obtain the distribution of the target hard disk and the target chassis in the current link; wherein the current link is any link among all the links of the storage system, and any link in the storage system is a link formed by connecting all chassis to the same port of the main cabinet; there are at least two control nodes in the main cabinet; each of the chassis includes at least two controllers, and each of the chassis has at least one controller connected to the first control node in the main cabinet, and at least another controller connected to the second control node in the main cabinet, and at least one controller is connected to the first port of each hard disk, and the remaining controllers are connected to the second port of each hard disk, and the remaining controllers are all controllers of each chassis except the controller connected to the first port of each hard disk; the target hard disk is a hard disk with a single port detected by the control node, and the target chassis is a chassis with a single controller detected by the control node;

[0008] Determine the current fault module according to the distribution of the target hard disk and the target chassis in the current link; wherein the current fault module includes at least one of the hard disk, the controller and the chassis;

[0009] The currently faulty module is repaired using a pre-established fault repair strategy.

[0010] On the one hand, determining the current fault module according to the distribution of the target hard disk and the target chassis in the current link includes:

[0011] The current fault module is determined according to the number of target hard disks on the current link, the distribution of the chassis where the target hard disks are located, the target port status of each target hard disk and the status of the target chassis; the target port is the port that the control node fails to detect.

[0012] On the other hand, determining the target hard disk includes:

[0013] Obtaining a preset hard disk port redundancy rule; wherein the hard disk port redundancy rule is determined at least by the state of the slot port corresponding to the hard disk port, the state of the hard disk port logging into the control node, or the situation where the control node sends an input / output request to the hard disk port;

[0014] When it is detected that the current hard disk does not satisfy the hard disk port redundancy rule, the current hard disk is determined to be the target hard disk.

[0015] On the other hand, the hard disk port redundancy rule is: the status of the slot port corresponding to the hard disk port is open, the status of the hard disk port logging into the control node is normal login status, and the situation where the control node sends input and output requests to the hard disk port is that the control node normally sends input and output requests to the hard disk port.

[0016] On the other hand, determining the target chassis includes:

[0017] Obtaining a preset controller redundancy rule; wherein the controller redundancy rule is determined at least by the in-place status of the controller, the status of the controller logging into the control node, or the situation of the control node detecting and accessing the controller;

[0018] When it is detected that the current chassis does not satisfy the controller redundancy rule, the current chassis is determined to be the target chassis.

[0019] On the other hand, the controller redundancy rule is: the controller's in-place state is in-place state, the controller's login state of the control node is normal login state, and the control node's detection and access to the controller is that the control node detects and accesses the controller.

[0020] On the other hand, the current fault module is determined according to the number of target hard disks on the current link, the distribution of the chassis where the target hard disks are located, the target port status of the target hard disks, and the status of the target chassis, including:

[0021] In the case where it is detected that the number of target hard disks in the current link is less than or equal to a first preset number, determining that the current faulty module is the target hard disk;

[0022] When it is detected that the number of target chassis in the current link is less than or equal to a second preset number, the current faulty module is determined to be a target controller in the target chassis; wherein the target controller is a controller among all controllers of the target chassis except the controller detected by the control node.

[0023] On the other hand, the current fault module is determined according to the number of target hard disks on the current link, the distribution of the chassis where the target hard disks are located, the target port status of the target hard disks, and the status of the target chassis, including:

[0024] When it is detected that the number of the target hard disks is greater than a first preset number, the current faulty module is determined according to the distribution of the chassis where the target hard disks are located and the target port status of the target hard disks.

[0025] On the other hand, the determining of the current fault module according to the distribution of the chassis where each of the target hard disks is located and the target port status of each of the target hard disks includes:

[0026] When it is detected that the target hard disks are distributed in the same chassis and the target ports of the target hard disks are the same, determining that the current faulty module is a controller connected to the target port;

[0027] If the number of the target hard disks is less than or equal to a third preset number, when it is detected that the target hard disks are distributed in the same chassis and the target ports of the target hard disks are different, it is determined that the current faulty module is the target hard disk, or that the current faulty module is the controller connected to the target ports; wherein the third preset number is greater than the first preset number;

[0028] If the number of the target hard disks is greater than the third preset number, when it is detected that the target hard disks are distributed in the same chassis and the target ports of the target hard disks are different, it is determined that the current faulty module is the chassis where the target hard disks are located.

[0029] On the other hand, the determining of the current fault module according to the distribution of the chassis where each of the target hard disks is located and the target port status of each of the target hard disks includes:

[0030] When it is detected that the target hard disks are distributed in different chassis and the target ports of the target hard disks are the same, determining that the current fault module is a controller connected to the target ports;

[0031] If the number of the target hard disks is less than or equal to a third preset number, when it is detected that the target hard disks are distributed in different chassis and the target ports of the target hard disks are different, it is determined that the current faulty module is the target hard disk; wherein the third preset number is greater than the first preset number;

[0032] If the number of target hard disks is greater than the third preset number, when it is detected that the target hard disks are distributed in different chassis and the target ports of the target hard disks are different, it is determined that the current faulty module is the target hard disk, the controller connected to each target port, or the chassis where the target hard disks are located.

[0033] On the other hand, the current fault module is determined according to the number of target hard disks on the current link, the distribution of the chassis where the target hard disks are located, the target port status of the target hard disks, and the status of the target chassis, including:

[0034] In the case where the existence of the target hard disk and the target chassis is detected, it is determined that the current faulty module is the target hard disk, a controller connected to the target port, or a chassis where the target hard disk is located.

[0035] On the other hand, the repairing the current faulty module by using a pre-established fault repair strategy includes:

[0036] When it is detected that the current faulty module is a hard disk, closing the slot port corresponding to the target port of the target hard disk and opening the slot ports corresponding to all ports of the target hard disk to repair the target hard disk;

[0037] In the case where it is detected that the current faulty module is a controller failure connected to the target port, other controllers in the chassis where the controller connected to the target port is located are obtained and the controller connected to the target port is reset using the other controllers to repair the controller; wherein the other controllers are controllers other than the controller connected to the target port among all controllers in the chassis where the controller connected to the target port is located;

[0038] When it is detected that the current faulty module is a chassis, a prompt message indicating that the user needs to replace the chassis, controller or hard disk is sent to the user, so as to repair the chassis.

[0039] On the other hand, when it is detected that the number of target hard disks in the current link is less than or equal to the first preset number and it is determined that the current faulty module is the target hard disk, before closing the slot port corresponding to the target port of the target hard disk and opening the slot ports corresponding to all ports of the target hard disk, the method further includes:

[0040] Starting from determining that the current faulty module is the target hard disk, determining whether the target hard disk fault is continuously detected within a first preset time period;

[0041] If so, the process proceeds to the step of closing the slot port corresponding to the target port of the target hard disk and simultaneously opening the slot ports corresponding to all ports of the target hard disk.

[0042] On the other hand, after closing the slot port corresponding to the target port of the target hard disk and opening the slot ports corresponding to all ports of the target hard disk, the method further includes:

[0043] After simultaneously opening the slot ports corresponding to all ports of the target hard disk, obtaining the repair status of the target hard disk detected within a second preset time period;

[0044] In the case where it is detected that the target hard disk has not been repaired, determining whether a redundant array of independent disks is redundant;

[0045] If yes, the target hard disk is isolated for faults;

[0046] If not, a prompt message is sent to the user indicating that the target hard disk needs to be repaired by the user.

[0047] On the other hand, closing the slot port corresponding to the target port of the target hard disk and opening the slot ports corresponding to all ports of the target hard disk comprises:

[0048] If the current fault module is a hard disk fault determined under the first target scenario, the slot ports corresponding to the target ports of each target hard disk are closed in turn, and the slot ports corresponding to all ports of each target hard disk are opened at the same time; wherein, the first target scenario is that the number of target hard disks is greater than the first preset number, the number of target hard disks is less than or equal to the third preset number, the target hard disks are distributed in the same chassis, and the target ports of the target hard disks are different, and the third preset number is greater than the first preset number.

[0049] On the other hand, when it is detected that the number of target chassis in the current link is less than or equal to the second preset number and it is determined that the current faulty module is a controller in the target chassis, before resetting the controller connected to the target port by using the other controller, the method further includes:

[0050] Starting from determining that the current faulty module is the controller, determining whether a controller fault is continuously detected within a third preset time period;

[0051] If yes, the step of using the other controller to reset the controller connected to the target port is entered.

[0052] On the other hand, after resetting the controller connected to the target port by using the other controller, the method further includes:

[0053] Starting from resetting the controller connected to the target port by using the other controller, obtaining the repair status of the detection controller within a fourth preset time period; wherein the fourth preset time period is greater than the third preset time period;

[0054] In the case where it is detected that the controller is not repaired, determining whether a controller other than the controller connected to the target port in a chassis where the controller connected to the target port is located is redundant;

[0055] If so, shut down the port of the controller connected to the target port to perform fault isolation on the controller connected to the target port;

[0056] If not, a prompt message is sent to the user indicating that the controller connected to the target port needs to be repaired by the user.

[0057] On the other hand, obtaining other controllers in the chassis where the controller connected to the target port is located and resetting the controller connected to the target port by using the other controllers includes:

[0058] If the current fault module is a controller failure determined under a second target scenario, other controllers in the chassis where the controller connected to the target port is located are obtained in sequence, and the controller connected to the target port is reset using the other controllers in the corresponding chassis; wherein the second target scenario is that the number of target hard disks is greater than the first preset number, each of the target hard disks is distributed in different chassis, and the target ports of each of the target hard disks are the same.

[0059] In order to solve the above technical problem, the present invention further provides a storage system fault handling device, the device comprising:

[0060] An acquisition module is used to acquire the distribution of the target hard disk and the target chassis in the current link; wherein the current link is any link among all the links of the storage system, and any link in the storage system is a link formed by connecting all chassis to the same port of the main cabinet; there are at least two control nodes in the main cabinet; each of the chassis includes at least two controllers, and each of the chassis has at least one controller connected to the first control node in the main cabinet, and at least another controller connected to the second control node in the main cabinet, and at least one controller is connected to the first port of each hard disk, and the remaining controllers are connected to the second port of each hard disk, and the remaining controllers are controllers of all the controllers of each chassis except the controller connected to the first port of each hard disk; the target hard disk is a hard disk with a single port detected by the control node, and the target chassis is a chassis with a single controller detected by the control node;

[0061] A determination module, configured to determine a current fault module according to the distribution of the target hard disk and the target chassis in the current link; wherein the current fault module includes at least one of a hard disk, a controller and a chassis;

[0062] The repair module is used to repair the current fault module by using a pre-established fault repair strategy.

[0063] In order to solve the above technical problem, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the above storage system fault handling method when executed by a processor.

[0064] In order to solve the above technical problems, the present invention further provides a storage system, comprising:

[0065] Memory for storing computer programs;

[0066] A processor is used to implement the steps of the above-mentioned storage system fault handling method when executing the computer program.

[0067] In order to solve the above technical problem, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above storage system fault handling method are implemented.

[0068] The storage system fault handling method provided by the present invention includes the distribution of target hard disks and target chassis in the current link, determining the current fault module according to the distribution of target hard disks and target chassis in the current link, and then repairing the current fault module using a pre-established fault repair strategy.

[0069] The beneficial effects of the present invention are that, firstly, the current fault module is determined according to the distribution of the target hard disk (i.e., the hard disk with a single port detected by the control node) and the target chassis (i.e., the chassis with a single controller detected by the control node) in the current link, and then the current fault module is repaired by using the pre-established fault repair strategy. It can be seen that the reliability, reliability and serviceability of the storage system are improved by identifying the fault module and repairing the fault module; secondly, there are at least two control nodes in the main cabinet, and each chassis connected to the main cabinet includes at least two controllers, and there is at least one controller in each chassis connected to the first control node in the main cabinet, and at least another controller connected to the second control node in the main cabinet, and each port of each hard disk is connected to a corresponding controller, that is, the controller node redundancy setting, the controller redundancy setting and the hard disk port redundancy setting ensure the redundancy of the storage system and improve the reliability of the storage system; thirdly, in a redundant storage system, the link formed by connecting all chassis with the same port of the main cabinet as a detection unit is used as a detection unit, which improves the efficiency of fault handling compared to the method of using each chassis as a detection unit.

[0070] In addition, the hard disk port redundancy rules are pre-set, specifically, the slot port corresponding to the hard disk port is in the open state, the hard disk port logs into the control node normally, and the control node sends input and output requests to the hard disk port normally. The hard disk port redundancy rules are used to determine whether the hard disk port can provide redundancy capabilities; the controller redundancy rules are pre-set, specifically, the controller is in place, the controller logs into the control node normally, and the control node detects and accesses the controller. The controller redundancy rules are used to determine whether the controller can provide redundancy capabilities.

[0071] By judging that the number of target hard disks in the current link is less than or equal to a first preset number, a target hard disk failure is determined; by judging that the number of target chassis in the current link is less than or equal to a second preset number, a target controller failure in the target chassis is determined; by judging that the number of target hard disks is greater than the first preset number, the current faulty module is further determined based on the distribution of the chassis where the target hard disks are located and the target port status of each target hard disk.

[0072] When it is detected that the number of target hard disks is greater than the first preset number, the current fault module is determined according to the distribution of the chassis where each target hard disk is located and the target port of each target hard disk. When each target hard disk is distributed in the same chassis, the fault modules in different scenarios are determined respectively. For example, if the target ports of each target hard disk are the same, it is determined that the controller connected to the target port is faulty; if the number of target hard disks is less than or equal to the third preset number (the third preset number is greater than the first preset number) and the target ports of each target hard disk are different, it is determined that the target hard disk or the controller connected to each target port is faulty; if the number of target hard disks is greater than the third preset number and the target ports of each target hard disk are different, it is determined that the chassis where the target hard disk is located is faulty. At the same time, when each target hard disk is distributed in different chassis, the fault modules in different scenarios are also determined respectively. If the target ports of each target hard disk are the same, it is determined that the controller connected to each target port is faulty; if the number of target hard disks is less than or equal to the third preset number and the target ports of each target hard disk are different, it is determined that the target hard disk is faulty; if the number of target hard disks is greater than the third preset number and the target ports of each target hard disk are different, it is determined that the target hard disk, the controller connected to each target port or the chassis where each target hard disk is located is faulty. In addition, for a scenario where a target hard disk and a target chassis overlap, it is determined that the current faulty module is the target hard disk, a controller connected to the target port, or the chassis where the target hard disk is located.

[0073] By closing the slot port corresponding to the target port of the target hard disk and opening the slot ports corresponding to all ports of the target hard disk at the same time, the hard disk fault is repaired; before the repair, the hard disk fault is repaired only when the target hard disk fault is continuously detected within the first preset time, so that the hard disk fault is accurately judged and the hard disk is effectively repaired; after the repair, the hard disk repair result is further judged. If the hard disk is not repaired, the judgment of whether the independent disk redundant array is redundant is continued. If it is redundant, the faulty hard disk is isolated, otherwise, the user is prompted to repair it, so as to ensure the redundancy of the storage system.

[0074] The slot ports corresponding to the target ports of each target hard disk are closed in turn, and the slot ports corresponding to all ports of each target hard disk are opened at the same time, thereby realizing the repair of hard disk failures in the scenario where the number of target hard disks is greater than a first preset number, the number of target hard disks is less than or equal to a third preset number, the target hard disks are distributed in the same chassis, and the target ports of the target hard disks are different, thereby ensuring the redundancy of the storage system.

[0075] By obtaining other controllers in the chassis where the controller connected to the target port is located and using other controllers to reset the controller connected to the target port, the faulty controller is repaired; before the repair, the controller fault is repaired only when the controller fault is continuously detected within the third preset time period, thereby accurately judging the controller fault and effectively repairing the controller fault; after the repair, the result of the controller repair is further judged. If the controller is not repaired, the judgment is continued as to whether the controllers other than the controller connected to the target port in the chassis where the controller connected to the target port is located are redundant. If they are redundant, the faulty controller is isolated; otherwise, the user is prompted to repair the controller, thereby ensuring the redundancy of the storage system.

[0076] Other controllers in the chassis where the controller connected to the target port is located are obtained in sequence, and the controller connected to the target port is reset using the other controllers in the corresponding chassis, thereby realizing the repair of controller failures in scenarios where the number of target hard disks is greater than a first preset number, the target hard disks are distributed in different chassis, and the target ports of the target hard disks are the same, thereby ensuring the redundancy of the storage system.

[0077] In addition, the present invention also provides a storage system fault handling device, a computer program product, a storage system, and a computer-readable storage medium, which have the same or corresponding technical features as the above-mentioned storage system fault handling method and have the same effects as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0079] Figure 1 A flowchart of a storage system fault handling method provided by an embodiment of the present invention;

[0080] Figure 2 A schematic diagram of a storage system composed of a SAS network provided by an embodiment of the present invention;

[0081] Figure 3 A schematic diagram of enumerating all single-port scenarios provided by an embodiment of the present invention;

[0082] Figure 4 A process of a method for repairing a SPA fault scenario provided by an embodiment of the present invention;

[0083] Figure 5A flowchart of a method for repairing a CSPA fault scenario provided by an embodiment of the present invention;

[0084] Figure 6 A structural diagram of a storage system provided in another embodiment of the present invention. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] The core of the present invention is to provide a storage system fault handling method, device, product, storage system and medium to solve the technical problem that the storage system has data loss or service interruption, which affects user business.

[0087] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 A flowchart of a storage system fault handling method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0088] S10: Obtain the distribution of the target hard disk and the target chassis in the current link;

[0089] S11: determining the current faulty module according to the distribution of the target hard disk and the target chassis in the current link;

[0090] S12: Repair the current faulty module using a pre-established fault repair strategy.

[0091] The following first describes the structure of the storage system provided by the present invention by taking the example that the main cabinet includes two control nodes (since the expansion cabinet also includes a controller, in order to distinguish it from the controller in the main cabinet, the controller in the main cabinet is called a control node in the embodiment of the present invention) and each expansion cabinet includes two controllers. Figure 2A schematic diagram of a storage system composed of a SAS network provided in an embodiment of the present invention. The storage system includes a main cabinet and multiple expansion cabinets. The main cabinet is connected to several expansion cabinets via a serial attached SCSI (SAS) card cable, supports dual-port hard disks, and forms an input / output (IO) group. The entire system provides disk array services to the host. The main cabinet is composed of two controllers, each controller has several SAS cards, and the Fibre Channel Card (FibreChannel Card) in the main cabinet is connected to the host port. In an embodiment of the present invention, the same link (Chain) is used as a fault detection unit. Any link in the storage system is a link formed by connecting the same port of the main cabinet to all chassis. As Figure 2 As shown in the figure, the physical topology is that the upper and lower control SAS cards come out of the same port (PORT), and all the expansion cabinets and hard disks connected in a "hand-in-hand" manner are called the same link. The main cabinet is connected to 20 expansion cabinets (chassis) through the SAS network. The main cabinet and the expansion cabinet can access dual-port hard disks. SAS card port one connects expansion cabinets one to expansion cabinet ten by hand-in-hand, which can be considered to be 10 expansion cabinets connected in a chain through link one; SAS card port two connects expansion cabinets eleven to expansion cabinet twenty by hand-in-hand, which can be considered to be 10 expansion cabinets connected in a chain through link two.

[0092] In actual production activities, the storage system provides massive disk array services. The system is generally composed of a main cabinet connected to dozens of expansion cabinets through a SAS network. The main cabinet and the expansion cabinet support dual-port hard disks, and the entire system supports more than 1,000 hard disks. If a single or superimposed fault point occurs in the SAS network, a series of problems will occur, such as:

[0093] 1) The faulty module will cause packet loss in the SAS network.

[0094] The low SAS rate will cause retries of commands sent to the hard disk. These retries will seriously affect the IO performance of the storage system. In special scenarios, with these faulty modules, the node will not be able to communicate with the hard disk.

[0095] 2) SAS network periodic failures.

[0096] For example, a faulty controller needs to be restarted to recover, but every once in a while, the fault will reappear, which will also affect performance and may also cause complete inability to access the hard disk.

[0097] 3) Nodes and expansion cabinets interact through the SAS network.

[0098] The SAS discovery process is a complex interaction process between nodes, controllers, and hard disks. In extreme scenarios, the entire interaction process may get stuck and exit abnormally.

[0099] Therefore, it is very important to identify and repair the failure of the storage system. In order to ensure the reliability, availability and serviceability of the storage system, in the embodiment of the present invention, there are at least two control nodes in the main cabinet, and each chassis includes at least two controllers. In each chassis, there is at least one controller connected to the first control node in the main cabinet, and there is at least another controller connected to the second control node in the main cabinet, and at least one controller is connected to the first port of each hard disk, and the remaining controllers are connected to the second port of each hard disk. The remaining controllers are all controllers of each chassis except the controller connected to the first port of each hard disk. The control nodes in the main cabinet can interact with each other, and the controllers in the expansion cabinet can also interact with each other. That is, multiple control nodes are set in the main cabinet, and multiple controllers are set in each chassis. Both the main cabinet and the chassis can be connected to multiple dual-port hard disks, so that the storage system has redundancy.

[0100] In the entire SAS network, a single port of a controller or hard disk may have a single point of failure, or there may be overlapping failure points. For this complex scenario, the connected expansion cabinets and hard disks are grouped into links, and the same link in the storage system is used as an example for fault analysis. First, the distribution of the target hard disk (a hard disk with a single port detected by the control node, also known as a single-port visible hard disk) and the target chassis (a chassis with a single controller detected by the control node. From the perspective of the entire chassis, since only one controller is visible, it is also called a controller single port visible) in the current link is obtained. Any link in the storage system is a link formed by connecting all chassis with the same port of the main cabinet, and the current link is any link in all links of the storage system. The target hard disk is a hard disk with a single port detected by the control node, and the target chassis is a chassis with a single controller detected by the control node. It should be noted that since the connection relationship of the link is determined, the current fault module (such as a hard disk, a controller or a chassis) and the position of the current fault module in the link can be determined according to the distribution of the target hard disk and the target chassis in the current link. Specifically, the current fault module is determined according to the number of target hard disks on the current link, the distribution of the chassis where the target hard disks are located, the target port status of each target hard disk and the status of the target chassis; wherein the target port is a port that the control node fails to detect.

[0101] After the current faulty module is determined, the current faulty module is repaired according to the pre-established fault repair strategy. The fault repair strategy includes the repair strategy corresponding to the hard disk fault, the repair strategy corresponding to the controller fault, and the repair strategy corresponding to the chassis. There is no limitation on the established repair strategy, as long as the faulty module can be repaired or processed.

[0102] In the method provided in this embodiment, first, the current fault module is determined according to the distribution of the target hard disk (i.e., the hard disk with a single port detected by the control node) and the target chassis (i.e., the chassis with a single controller detected by the control node) in the current link, and then the current fault module is repaired using the pre-established fault repair strategy. It can be seen that the reliability, reliability and serviceability of the storage system are improved by identifying the fault module and repairing the fault module; secondly, there are at least two control nodes in the main cabinet, and each chassis connected to the main cabinet includes at least two controllers, and there is at least one controller in each chassis connected to the first control node in the main cabinet, and at least another controller connected to the second control node in the main cabinet, and each port of each hard disk is connected to a corresponding controller, that is, the controller node redundancy setting, the controller redundancy setting and the hard disk port redundancy setting ensure the redundancy of the storage system and improve the reliability of the storage system; thirdly, in a redundant storage system, the link formed by connecting all chassis with the same port of the main cabinet as a detection unit is used as a detection unit, which improves the efficiency of fault handling compared to the method of using each chassis as a detection unit.

[0103] The target hard disk is a hard disk detected by the control node on a single port. In implementation, determining the target hard disk includes:

[0104] Obtaining a preset hard disk port redundancy rule; wherein the hard disk port redundancy rule is determined by at least the state of the slot port corresponding to the hard disk port, the state of the hard disk port logging into the control node, or the situation where the control node sends an input / output request to the hard disk port;

[0105] When it is detected that the current hard disk does not meet the hard disk port redundancy rule, the current hard disk is determined to be the target hard disk.

[0106] Specifically, the hard disk port redundancy rules are: the status of the slot port corresponding to the hard disk port is open, the status of the hard disk port logging into the control node is normal, and the situation where the control node sends input and output requests to the hard disk port is that the control node normally sends input and output requests to the hard disk port.

[0107] Hard disk port redundancy means that both ports of the hard disk can be accessed by the corresponding nodes. Disk port redundancy means that port 1 can provide redundancy to port 2, that is, when port 2 cannot work due to a fault, port 1 can still complete the IO delivery function required by the storage system. In view of this requirement, the following rules are designed taking port 1 as an example:

[0108] 1) The hard disk port is open as soon as it is inserted into the corresponding slot.

[0109] 2) The login to hard disk port 1 is normal and can log in to the corresponding control node 1 normally.

[0110] 3) Control node 1 can send IO to hard disk port 1 normally.

[0111] When this rule is met, port 1 is considered to be able to provide redundancy to port 2. Similarly, if port 2 can also provide redundancy to port 1, the two ports of this hard disk are considered to be redundant. When one port does not meet the rule, the hard disk is considered to be single port active (SPA).

[0112] The target chassis is a chassis where the control node detects a single controller. In an implementation, determining the target chassis includes:

[0113] Obtaining a preset controller redundancy rule; wherein the controller redundancy rule is determined at least by the in-place status of the controller, the status of the controller logging into the control node, or the situation of the control node detecting and accessing the controller;

[0114] When it is detected that the current chassis does not meet the controller redundancy rule, the current chassis is determined to be the target chassis.

[0115] Specifically, the controller redundancy rule is: the controller in-place state is in-place state, the controller login control node state is login normal state, and the control node detection and access to the controller is the control node detection and access to the controller.

[0116] Controller redundancy means that both controllers of the chassis can be accessed by the corresponding nodes. The controller redundancy rule is that controller 1 can provide redundancy to controller 2, that is, when controller 2 fails to work, controller 1 can still complete the IO delivery function required by the storage system. In view of this requirement, the following rules are designed taking controller 1 as an example:

[0117] 1) The controller is in place, that is, the controller is inserted into the chassis.

[0118] 2) The login of controller one is normal and can log in to the corresponding control node one normally.

[0119] 3) Controller Node 1 can see Controller 1 and can access it through the SAS network.

[0120] When this rule is met, controller 1 is considered to provide redundancy to controller 2. Similarly, if controller 2 can also provide redundancy to controller 1, the two controllers in the chassis are considered to be redundant. When one controller does not meet this rule, it is considered that only one controller single port is visible (Canister SPA, CSPA).

[0121] In the method provided in this embodiment, the hard disk port redundancy rules are pre-set, specifically, the slot port corresponding to the hard disk port is in an open state, the hard disk port logs into the control node normally, and the control node sends input and output requests to the hard disk port normally. The hard disk port redundancy rules are used to determine whether the hard disk port can provide redundancy capabilities; the pre-set controller redundancy rules are specifically, the controller is in place, the controller logs into the control node normally, and the control node detects and accesses the controller. The controller redundancy rules are used to determine whether the controller can provide redundancy capabilities.

[0122] The storage system composed of the entire SAS network contains dozens of expansion cabinets and more than 1,000 hard disks. CSPA and SPA can appear anywhere on the network and can also appear in superposition, so single-port scenarios appear in an endless stream. However, starting from the physical topology of the storage system, if it is grouped by link, it is found that it will simplify the identification of faulty modules, because the single-port scenarios that appear in each group of links are in the same limited set. According to the number of target hard disks on the current link, the distribution of the chassis where each target hard disk is located, the target port status of each target hard disk, and the status of the target chassis, the current faulty module is determined to include:

[0123] When it is detected that the number of target hard disks in the current link is less than or equal to the first preset number, the current faulty module is determined to be the target hard disk. This scenario is called a SPA scenario.

[0124] When it is detected that the number of target chassis in the current link is less than or equal to the second preset number, it is determined that the current faulty module is a target controller in the target chassis; wherein the target controller is a controller among all controllers of the target chassis except the controller detected by the control node. This scenario is called a CSPA scenario.

[0125] When it is detected that the number of target hard disks is greater than the first preset number, the current faulty module is determined according to the distribution of chassis where the target hard disks are located and the target port status of the target hard disks.

[0126] There is no limitation on the first preset number and the second preset number, for example, both are 1.

[0127] When the target hard disks are distributed in the same chassis, the current fault module is determined according to the distribution of the chassis where the target hard disks are located and the target port status of the target hard disks, including:

[0128] When it is detected that the target hard disks are distributed in the same chassis and the target ports of the target hard disks are the same, it is determined that the current faulty module is the controller connected to the target port. This scenario is called a Multiple SPA (MSPA) scenario.

[0129] If the number of target hard disks is less than or equal to the third preset number, when it is detected that the target hard disks are distributed in the same chassis and the target ports of the target hard disks are different, it is determined that the current faulty module is the target hard disk, or the current faulty module is determined to be the controller connected to each target port; wherein the third preset number is greater than the first preset number. This scenario is called a Dual SPA (DSPA) scenario.

[0130] If the number of target hard disks is greater than the third preset number, and it is detected that the target hard disks are distributed in the same chassis and the target ports of the target hard disks are different, it is determined that the current faulty module is the chassis where the target hard disks are located. This scenario is called a Dual Multiple SPA (DMSPA) scenario.

[0131] There is no limitation on the third preset number, as long as it is greater than the first preset number, for example, the third preset number is 2.

[0132] When the target hard disks are distributed in different chassis, the current fault module is determined according to the distribution of the chassis where the target hard disks are located and the target port status of the target hard disks, including:

[0133] When it is detected that the target hard disks are distributed in different enclosures and the target ports of the target hard disks are the same, it is determined that the current faulty module is the controller connected to the target ports. This scenario is called Multiple Enclosures SPA (MESPA) scenario.

[0134] If the number of target hard disks is less than or equal to the third preset number, and it is detected that the target hard disks are distributed in different chassis and the target ports of the target hard disks are different, it is determined that the current faulty module is the target hard disk; wherein the third preset number is greater than the first preset number. This scenario is called the Dual Enclosures SPA (DESPA) scenario.

[0135] If the number of target hard disks is greater than the third preset number, and it is detected that the target hard disks are distributed in different chassis and the target ports of the target hard disks are different, it is determined that the current faulty module is the target hard disk, the controller connected to each target port, or the chassis where each target hard disk is located. This scenario is called Dual Multiple Enclosures SPA (DMESPA) scenario.

[0136] In the scenario where a single hard disk port is visible and a single controller port is visible, the current fault module is determined based on the number of target hard disks on the current link, the distribution of the chassis where the target hard disks are located, the target port status of each target hard disk, and the status of the target chassis. The modules include:

[0137] When the target hard disk and the target chassis are detected, it is determined that the current faulty module is the target hard disk, the controller connected to the target port, or the chassis where the target hard disk is located. This scenario is called the CSPA_PLUS scenario.

[0138] Table 1 is an exhaustive list of single-port sets under a set of links. Figure 3 A schematic diagram of enumerating all single-port scenarios provided by an embodiment of the present invention. Figure 3 As shown, the nine single-port scenarios in Table 1 are shown.

[0139] Table 1

[0140]

[0141] Various fault scenarios are enumerated above. In order to handle the faulty module, the pre-established fault repair strategy is used to repair the current faulty module, including:

[0142] When it is detected that the current faulty module is a hard disk, the slot port corresponding to the target port of the target hard disk is closed, and the slot ports corresponding to all ports of the target hard disk are opened to repair the target hard disk.

[0143] When it is detected that the current faulty module is a controller connected to the target port, other controllers in the chassis where the controller connected to the target port is located are obtained and the controller connected to the target port is reset using the other controllers to repair the controller; wherein the other controllers are controllers other than the controller connected to the target port among all controllers in the chassis where the controller connected to the target port is located.

[0144] When it is detected that the current faulty module is a chassis, a prompt message is sent to the user indicating that the user needs to replace the chassis, controller or hard disk, so as to repair the chassis.

[0145] Based on the content described above, the fault points are classified into hard disk, controller and chassis faults. For each type of faulty module, a corresponding repair strategy is designed according to the hardware attributes. It is particularly noted that for chassis faults, the software cannot attempt to repair them, and the specific link number can be reported to prompt the user to intervene and replace the chassis, controller or hard disk.

[0146] In practice, after determining that the hard disk fails, the hard disk may only be temporarily faulty. Therefore, in order to effectively handle the hard disk failure, when it is detected that the number of target hard disks in the current link is less than or equal to the first preset number, it is determined that the current faulty module is the target hard disk, before closing the slot port corresponding to the target port of the target hard disk and opening the slot ports corresponding to all ports of the target hard disk, the method further includes:

[0147] Starting from determining that the current faulty module is the target hard disk, determining whether the target hard disk fault is continuously detected within a first preset time period;

[0148] If so, the step of closing the slot port corresponding to the target port of the target hard disk and opening the slot ports corresponding to all ports of the target hard disk is entered.

[0149] There is no limitation on the first preset time length, such as 30 seconds.

[0150] The faulty hard disk is recovered by closing the slot port corresponding to the target port of the target hard disk and opening the slot ports corresponding to all ports of the target hard disk. However, in order to determine the recovery result of the faulty hard disk, after closing the slot port corresponding to the target port of the target hard disk and opening the slot ports corresponding to all ports of the target hard disk, the faulty hard disk is also recovered:

[0151] After the slot ports corresponding to all ports of the target hard disk are opened simultaneously, the repair status of the target hard disk is detected within a second preset time period;

[0152] In the case where it is detected that the target hard disk has not been repaired, determining whether the redundant array of independent disks is redundant;

[0153] If yes, the target hard disk is isolated for fault;

[0154] If not, a prompt message is sent to the user indicating that the target hard disk is to be repaired by the user.

[0155] There is no limitation on the second preset time length, such as 30 seconds.

[0156] Figure 4 A flowchart of a method for repairing a SPA fault scenario provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the method includes:

[0157] S13: Determining that only one disk port is visible;

[0158] S14: Determine whether the timer has reached 30s; if not, return to step S13; if yes, proceed to step S15;

[0159] S15: Collect the controller firmware log corresponding to the disk;

[0160] S16: close the slot port of the single-port visible disk;

[0161] S17: Open two slot ports at the same time;

[0162] S18: Start a 30s timer;

[0163] S19: Determine whether the single-port visible disk is repaired; if so, end; if not, proceed to step S20;

[0164] S20: Determine whether the redundant array of independent disks is redundant; if so, proceed to step S21; if not, proceed to step S22;

[0165] S21: Remove the hard disk and isolate the fault.

[0166] S22: Report an alarm.

[0167] In the method provided in this embodiment, the hard disk fault is repaired by closing the slot port corresponding to the target port of the target hard disk and opening the slot ports corresponding to all ports of the target hard disk at the same time; before the repair, the hard disk fault is repaired only when the target hard disk fault is continuously detected within the first preset time, thereby accurately judging the hard disk fault and effectively repairing the hard disk; after the repair, the hard disk repair result is further judged. If the hard disk is not repaired, the judgment of whether the independent disk redundant array is redundant is continued. If it is redundant, the faulty hard disk is isolated, otherwise, the user is prompted to repair it, thereby ensuring the redundancy of the storage system.

[0168] The first target scenario is that the number of target hard disks is greater than the first preset number, the number of target hard disks is less than or equal to the third preset number, the target hard disks are distributed in the same chassis, and the target ports of the target hard disks are different, and the third preset number is greater than the first preset number (i.e., the DSPA scenario described above). The current fault module is a hard disk fault determined in the first target scenario, closing the slot port corresponding to the target port of the target hard disk, and opening the slot ports corresponding to all ports of the target hard disk include:

[0169] If the current fault module is a hard disk fault determined in the first target scenario, the slot ports corresponding to the target ports of each target hard disk are closed in sequence, and the slot ports corresponding to all ports of each target hard disk are opened at the same time.

[0170] That is, for the DSPA failure scenario, it is only necessary to run the SPA repair strategy multiple times to achieve the purpose of repairing the faulty hard disk in the DSPA failure scenario.

[0171] In practice, after determining that the controller fails, the controller may only be temporarily faulty. Therefore, in order to effectively handle the controller failure, when it is detected that the number of target chassis in the current link is less than or equal to the second preset number and it is determined that the current faulty module is the controller in the target chassis, before resetting the controller connected to the target port by using other controllers, the method further includes:

[0172] Starting from determining that the current faulty module is the controller, determining whether the controller fault is continuously detected within a third preset time period;

[0173] If yes, the process proceeds to the step of resetting the controller connected to the target port by using another controller.

[0174] There is no limitation on the third preset time length, such as 30 seconds.

[0175] The faulty controller is recovered by resetting the controller connected to the target port using other controllers. However, in order to determine the recovery result of the faulty controller, after resetting the controller connected to the target port using other controllers, the following steps are further included:

[0176] Starting from resetting the controller connected to the target port by using another controller, obtaining the repair status of the detection controller within a fourth preset time period; wherein the fourth preset time period is greater than the third preset time period;

[0177] In the case where it is detected that the controller is not repaired, determining whether a controller other than the controller connected to the target port in a chassis where the controller connected to the target port is located is redundant;

[0178] If so, shut down the port of the controller connected to the target port to isolate the controller connected to the target port;

[0179] If not, a prompt message is sent to the user indicating that the controller connected to the target port needs to be repaired by the user.

[0180] There is no limitation on the fourth preset time length, such as 90 seconds.

[0181] Figure 5 A flowchart of a method for repairing a CSPA fault scenario provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the method includes:

[0182] S23: Determining that only one controller is visible;

[0183] S24: Determine whether the timer has reached 30 seconds; if not, return to step S23; if yes, proceed to step S25;

[0184] S25: Collect the firmware log of the faulty controller through the peer controller;

[0185] S26: Reset the firmware of the faulty controller through the peer controller;

[0186] S27: Start a 90s timer;

[0187] S28: Determine whether the controller is repaired; if so, end; if not, proceed to step S29;

[0188] S29: Determine whether the remaining controllers are redundant; if so, proceed to step S30; if not, proceed to step S31;

[0189] S30: close the faulty controller port and isolate the fault;

[0190] S31: Report an alarm.

[0191] In the method provided in this embodiment, the faulty controller is repaired by obtaining other controllers in the chassis where the controller connected to the target port is located and using other controllers to reset the controller connected to the target port; before the repair, the controller fault is repaired only when the controller fault is continuously detected within the third preset time period, thereby accurately judging the controller fault and effectively repairing the controller fault; after the repair, the result of the controller repair is further judged. If the controller is not repaired, the judgment of whether the controllers other than the controller connected to the target port in the chassis where the controller connected to the target port is located are redundant is continued. If they are redundant, the faulty controller is isolated; otherwise, the user is prompted to repair it, thereby ensuring the redundancy of the storage system.

[0192] The second target scenario is that the number of target hard disks is greater than the first preset number, the target hard disks are distributed in different chassis, and the target ports of the target hard disks are the same, that is, the MESPA scenario described above. The current fault module is the controller fault determined in the second target scenario, and obtaining other controllers in the chassis where the controller connected to the target port is located and using other controllers to reset the controller connected to the target port includes:

[0193] Other controllers in the chassis where the controller connected to the target port is located are obtained in sequence, and the controller connected to the target port is reset by using other controllers in the corresponding chassis.

[0194] That is, for the MESPA fault scenario, it is only necessary to run the CSPA repair strategy multiple times to achieve the purpose of repairing the faulty controller in the MESPA fault scenario.

[0195] It can be seen that in the fault handling method provided by the present invention, the controller and hard disk redundancy rules are used to further introduce the non-redundant concept of the controller single port and the hard disk single port. In the entire SAS network, the controller and hard disk single port may have single-point failures, and it is also possible that the fault points are superimposed. For this complex scenario, the connected expansion cabinets and hard disks are grouped by links, and the variables (including hard disks, controllers, and chassis) are identified. All single-port scenarios that appear are enumerated according to the variables. Then, the fault module and repair strategy are summarized according to each scenario, and then the corresponding repair process is formulated for each single-port scenario that is exhaustively enumerated, and the repair is performed in groups of links to achieve the purpose of automatically repairing the entire system. When periodic repair cannot be performed or hardware failure occurs, that is, a single-port scenario is detected, and after running the repair strategy, the fault is successfully repaired, but after a while, the scene where the fault occurs again, for this phenomenon, a rule is set. If it occurs 5 times in 24 hours, it will be considered that it cannot be automatically repaired, the fault module is isolated, and an alarm is reported to prompt the operation and maintenance personnel to intervene in the repair. The method can effectively detect whether the storage system is redundant, and automatically repair it. If the repair is unsuccessful, the fault module is isolated, an alarm is reported, and the customer service personnel are prompted to intervene in the repair. The redundancy of the entire system can be improved by this method.

[0196] In the above embodiments, the fault handling method of the storage system is described in detail, and the present invention also provides a fault handling device for the storage system and a corresponding embodiment of the storage system. It should be noted that the present invention describes the embodiments of the device part from two perspectives, one is based on the perspective of the functional module, and the other is based on the perspective of the hardware.

[0197] An embodiment of the present invention provides a storage system fault handling device. This embodiment is based on the perspective of functional modules and includes:

[0198] An acquisition module is used to obtain the distribution of the target hard disk and the target chassis in the current link; wherein the current link is any link among all the links of the storage system, and any link in the storage system is a link formed by connecting all chassis to the same port of the main cabinet; there are at least two control nodes in the main cabinet; each chassis includes at least two controllers, and each chassis has at least one controller connected to the first control node in the main cabinet, and at least another controller connected to the second control node in the main cabinet, and at least one controller is connected to the first port of each hard disk, and the remaining controllers are the controllers of all the controllers of each chassis except the controllers connected to the first port of each hard disk; the target hard disk is a hard disk with a single port detected by the control node, and the target chassis is a chassis with a single controller detected by the control node;

[0199] A determination module, used to determine the current fault module according to the distribution of the target hard disk and the target chassis in the current link; wherein the current fault module includes at least one of the hard disk, the controller and the chassis;

[0200] The repair module is used to repair the current fault module using a pre-established fault repair strategy.

[0201] In some embodiments, the determination module specifically includes a determination submodule;

[0202] The determination submodule is used to determine the current fault module according to the number of target hard disks on the current link, the distribution of the chassis where each target hard disk is located, the target port status of each target hard disk and the status of the target chassis; the target port is the port that the control node cannot detect.

[0203] In some embodiments, the fault handling device of the storage system further includes:

[0204] A first judgment module is used to judge whether the target hard disk failure is continuously detected within a first preset time period after the current faulty module is determined to be the target hard disk; if so, trigger the closing and opening modules;

[0205] The closing and opening module is used to close the slot port corresponding to the target port of the target hard disk, and to open the slot ports corresponding to all ports of the target hard disk.

[0206] In some embodiments, the fault handling device of the storage system further includes:

[0207] A first acquisition module is used to acquire the repair status of the target hard disk within a second preset time period after the slot ports corresponding to all ports of the target hard disk are opened simultaneously;

[0208] A second judgment module is used to judge whether the independent disk redundant array is redundant when it is detected that the target hard disk is not repaired; if so, trigger the first isolation module; if not, trigger the first sending module;

[0209] A first isolation module, used for isolating the target hard disk from failure;

[0210] The first sending module is used to send prompt information to the user for indicating the target hard disk to be repaired by the user.

[0211] In some embodiments, the fault handling device of the storage system further includes:

[0212] A third judgment module is used to judge whether the controller fault is continuously detected within a third preset time period after determining that the current faulty module is the controller; if so, trigger the reset module;

[0213] The reset module is used to reset the controller connected to the target port by using other controllers.

[0214] In some embodiments, the fault handling device of the storage system further includes:

[0215] A second acquisition module is used to acquire the repair status of the detection controller within a fourth preset time period starting from resetting the controller connected to the target port by using other controllers; wherein the fourth preset time period is greater than the third preset time period;

[0216] A fourth judgment module is used to judge whether the controller other than the controller connected to the target port in the chassis where the controller connected to the target port is located is redundant when it is detected that the controller is not repaired; if so, trigger the second isolation module; if not, trigger the second sending module;

[0217] A second isolation module, used for shutting down the port of the controller connected to the target port, so as to perform fault isolation on the controller connected to the target port;

[0218] The second sending module is used to send prompt information to the user, which is used to indicate that the controller connected to the target port needs to be repaired by the user.

[0219] Since the embodiments of the device part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the device part, and they will not be repeated here. The effects are the same as above.

[0220] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned storage system fault handling method when executed by a processor.

[0221] Figure 6 This is a structural diagram of a storage system provided by another embodiment of the present invention. This embodiment is based on the hardware perspective, such as Figure 6 As shown, the storage system includes:

[0222] A memory 20, for storing computer programs;

[0223] The processor 21 is configured to implement the steps of the method for handling a storage system failure as mentioned in the above embodiment when executing a computer program.

[0224] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0225] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the storage system fault handling method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the above-mentioned storage system fault handling method, etc.

[0226] In some embodiments, the storage system may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0227] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation of the storage system, and may include more or fewer components than shown in the figure.

[0228] The storage system provided by the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the following method: a fault handling method of the storage system, and the effect is the same as above.

[0229] Finally, the present invention also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.

[0230] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.

[0231] The computer-readable storage medium provided by the present invention includes the above-mentioned storage system fault handling method, and the effect is the same as above.

[0232] The above is a detailed introduction to the storage system fault handling method, device, product, storage system and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

[0233] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A method for handling a storage system failure, characterized in that: The method comprises: Obtain the distribution of the target hard disk and the target chassis in the current link; wherein the current link is any link among all the links of the storage system, and any link in the storage system is a link formed by connecting all chassis to the same port of the main cabinet; there are at least two control nodes in the main cabinet; each of the chassis includes at least two controllers, and each of the chassis has at least one controller connected to the first control node in the main cabinet, and at least another controller connected to the second control node in the main cabinet, and at least one controller is connected to the first port of each hard disk, and the remaining controllers are connected to the second port of each hard disk, and the remaining controllers are all controllers of each chassis except the controller connected to the first port of each hard disk; the target hard disk is a hard disk with a single port detected by the control node, and the target chassis is a chassis with a single controller detected by the control node; Determine the current fault module according to the distribution of the target hard disk and the target chassis in the current link; wherein the current fault module includes at least one of the hard disk, the controller and the chassis; Repairing the current faulty module using a pre-established fault repair strategy; The method of determining the current fault module according to the distribution of the target hard disk and the target chassis in the current link includes: Determine the current fault module according to the number of target hard disks on the current link, the distribution of the chassis where the target hard disks are located, the target port status of the target hard disks or the status of the target chassis; the target port is a port that the control node fails to detect; The current fault module is determined according to the number of target hard disks on the current link, the distribution of chassis where the target hard disks are located, the target port status of the target hard disks or the status of the target chassis, including: In the case where it is detected that the number of target hard disks in the current link is less than or equal to a first preset number, determining that the current faulty module is the target hard disk; In the case where it is detected that the number of target chassis in the current link is less than or equal to a second preset number, determining that the current faulty module is a target controller in the target chassis; wherein the target controller is a controller among all controllers of the target chassis except the controller detected by the control node; In the case where it is detected that the number of the target hard disks is greater than a first preset number, the current fault module is determined according to the distribution of the chassis where the target hard disks are located and the target port of each target hard disk; in the case where the target hard disk and the target chassis are detected, the current fault module is determined to be the target hard disk, the controller connected to the target port, or the chassis where the target hard disk is located; The repairing of the current fault module by using a pre-established fault repair strategy includes: When it is detected that the current faulty module is a hard disk, closing the slot port corresponding to the target port of the target hard disk and opening the slot ports corresponding to all ports of the target hard disk to repair the target hard disk; In the case where it is detected that the current faulty module is a controller failure connected to the target port, other controllers in the chassis where the controller connected to the target port is located are obtained and the controller connected to the target port is reset using the other controllers to repair the controller; wherein the other controllers are controllers other than the controller connected to the target port among all controllers in the chassis where the controller connected to the target port is located; When it is detected that the current faulty module is a chassis, a prompt message is sent to the user indicating that the user needs to replace the chassis, controller or hard disk, so as to repair the chassis; Wherein, the determining of the current fault module according to the distribution of the chassis where each of the target hard disks is located and the target port status of each of the target hard disks includes: When it is detected that the target hard disks are distributed in the same chassis and the target ports of the target hard disks are the same, determining that the current faulty module is a controller connected to the target port; If the number of the target hard disks is less than or equal to a third preset number, when it is detected that the target hard disks are distributed in the same chassis and the target ports of the target hard disks are different, it is determined that the current faulty module is the target hard disk, or that the current faulty module is the controller connected to the target ports; wherein the third preset number is greater than the first preset number; If the number of the target hard disks is greater than the third preset number, and it is detected that the target hard disks are distributed in the same chassis and the target ports of the target hard disks are different, it is determined that the current fault module is the chassis where the target hard disks are located; When it is detected that the target hard disks are distributed in different chassis and the target ports of the target hard disks are the same, determining that the current fault module is a controller connected to the target ports; If the number of the target hard disks is less than or equal to a third preset number, when it is detected that the target hard disks are distributed in different chassis and the target ports of the target hard disks are different, it is determined that the current faulty module is the target hard disk; wherein the third preset number is greater than the first preset number; If the number of target hard disks is greater than the third preset number, when it is detected that the target hard disks are distributed in different chassis and the target ports of the target hard disks are different, it is determined that the current faulty module is the target hard disk, the controller connected to each target port, or the chassis where the target hard disks are located.

2. The storage system fault handling method according to claim 1, characterized in that: Determining the target hard disk includes: Obtaining a preset hard disk port redundancy rule; wherein the hard disk port redundancy rule is determined at least by the state of the slot port corresponding to the hard disk port, the state of the hard disk port logging into the control node, or the situation where the control node sends an input / output request to the hard disk port; When it is detected that the current hard disk does not satisfy the hard disk port redundancy rule, the current hard disk is determined to be the target hard disk.

3. The storage system fault handling method according to claim 2, characterized in that: The hard disk port redundancy rule is: the state of the slot port corresponding to the hard disk port is open, the state of the hard disk port logging into the control node is normal, and the situation where the control node sends input and output requests to the hard disk port is that the control node normally sends input and output requests to the hard disk port.

4. The storage system fault handling method according to claim 1, characterized in that: Determining the target chassis includes: Obtaining a preset controller redundancy rule; wherein the controller redundancy rule is determined at least by the in-place status of the controller, the status of the controller logging into the control node, or the situation of the control node detecting and accessing the controller; When it is detected that the current chassis does not satisfy the controller redundancy rule, the current chassis is determined to be the target chassis.

5. The storage system fault handling method according to claim 4, characterized in that: The controller redundancy rule is: the controller's in-place state is in-place state, the controller's login state of the control node is normal login state, and the control node's detection and access to the controller is that the control node detects and accesses the controller.

6. The storage system fault handling method according to claim 1, characterized in that: When it is detected that the number of target hard disks in the current link is less than or equal to the first preset number and it is determined that the current faulty module is the target hard disk, before closing the slot port corresponding to the target port of the target hard disk and opening the slot ports corresponding to all ports of the target hard disk, the method further includes: Starting from determining that the current faulty module is the target hard disk, determining whether the target hard disk fault is continuously detected within a first preset time period; If so, the process proceeds to the step of closing the slot port corresponding to the target port of the target hard disk and simultaneously opening the slot ports corresponding to all ports of the target hard disk.

7. The storage system fault handling method according to claim 6, characterized in that: After closing the slot port corresponding to the target port of the target hard disk and opening the slot ports corresponding to all ports of the target hard disk, the method further includes: After simultaneously opening the slot ports corresponding to all ports of the target hard disk, obtaining the repair status of the target hard disk detected within a second preset time period; In the case where it is detected that the target hard disk has not been repaired, determining whether a redundant array of independent disks is redundant; If yes, the target hard disk is isolated for faults; If not, a prompt message is sent to the user indicating that the target hard disk needs to be repaired by the user.

8. The storage system fault handling method according to claim 1, characterized in that: The closing of the slot port corresponding to the target port of the target hard disk and the opening of the slot ports corresponding to all ports of the target hard disk comprises: If the current fault module is a hard disk fault determined under the first target scenario, the slot ports corresponding to the target ports of each target hard disk are closed in turn, and the slot ports corresponding to all ports of each target hard disk are opened at the same time; wherein, the first target scenario is that the number of target hard disks is greater than the first preset number, the number of target hard disks is less than or equal to the third preset number, the target hard disks are distributed in the same chassis, and the target ports of the target hard disks are different, and the third preset number is greater than the first preset number.

9. The storage system fault handling method according to claim 1, characterized in that: In the case where it is determined that the current faulty module is a controller in the target chassis when it is detected that the number of target chassis in the current link is less than or equal to the second preset number, before resetting the controller connected to the target port by using the other controller, the method further includes: Starting from determining that the current faulty module is the controller, determining whether a controller fault is continuously detected within a third preset time period; If yes, the step of using the other controller to reset the controller connected to the target port is entered.

10. The storage system fault handling method according to claim 9, characterized in that: After resetting the controller connected to the target port by using the other controller, the method further includes: Starting from resetting the controller connected to the target port by using the other controller, obtaining the repair status of the detection controller within a fourth preset time period; wherein the fourth preset time period is greater than the third preset time period; In the case where it is detected that the controller is not repaired, determining whether a controller other than the controller connected to the target port in a chassis where the controller connected to the target port is located is redundant; If so, shut down the port of the controller connected to the target port to perform fault isolation on the controller connected to the target port; If not, a prompt message is sent to the user indicating that the controller connected to the target port needs to be repaired by the user.

11. The storage system fault handling method according to claim 1, characterized in that: The acquiring of other controllers in the chassis where the controller connected to the target port is located and using the other controllers to reset the controller connected to the target port comprises: If the current fault module is a controller failure determined under a second target scenario, other controllers in the chassis where the controller connected to the target port is located are obtained in sequence, and the controller connected to the target port is reset using the other controllers in the corresponding chassis; wherein the second target scenario is that the number of target hard disks is greater than the first preset number, each of the target hard disks is distributed in different chassis, and the target ports of each of the target hard disks are the same.

12. A storage system fault handling device, characterized in that: The device comprises: An acquisition module is used to acquire the distribution of the target hard disk and the target chassis in the current link; wherein the current link is any link among all the links of the storage system, and any link in the storage system is a link formed by connecting all chassis to the same port of the main cabinet; there are at least two control nodes in the main cabinet; each of the chassis includes at least two controllers, and each of the chassis has at least one controller connected to the first control node in the main cabinet, and at least another controller connected to the second control node in the main cabinet, and at least one controller is connected to the first port of each hard disk, and the remaining controllers are connected to the second port of each hard disk, and the remaining controllers are controllers of all the controllers of each chassis except the controller connected to the first port of each hard disk; the target hard disk is a hard disk with a single port detected by the control node, and the target chassis is a chassis with a single controller detected by the control node; A determination module, configured to determine a current fault module according to the distribution of the target hard disk and the target chassis in the current link; wherein the current fault module includes at least one of a hard disk, a controller and a chassis; A repair module, used to repair the current fault module using a pre-established fault repair strategy; A determination module, specifically used to determine the current fault module according to the number of target hard disks on the current link, the distribution of the chassis where the target hard disks are located, the target port status of the target hard disks or the status of the target chassis; the target port is a port that the control node fails to detect; The current fault module is determined according to the number of target hard disks on the current link, the distribution of chassis where the target hard disks are located, the target port status of the target hard disks or the status of the target chassis, including: In the case where it is detected that the number of target hard disks in the current link is less than or equal to a first preset number, determining that the current faulty module is the target hard disk; In the case where it is detected that the number of target chassis in the current link is less than or equal to a second preset number, determining that the current faulty module is a target controller in the target chassis; wherein the target controller is a controller among all controllers of the target chassis except the controller detected by the control node; When it is detected that the number of the target hard disks is greater than a first preset number, determining the current faulty module according to the distribution of the chassis where the target hard disks are located and the target port status of the target hard disks; In the case where the target hard disk and the target chassis are detected, determining that the current faulty module is the target hard disk, a controller connected to the target port, or a chassis where the target hard disk is located; The repair module is specifically used for: When it is detected that the current faulty module is a hard disk, closing the slot port corresponding to the target port of the target hard disk and opening the slot ports corresponding to all ports of the target hard disk to repair the target hard disk; In the case where it is detected that the current faulty module is a controller failure connected to the target port, other controllers in the chassis where the controller connected to the target port is located are obtained and the controller connected to the target port is reset using the other controllers to repair the controller; wherein the other controllers are controllers other than the controller connected to the target port among all controllers in the chassis where the controller connected to the target port is located; When it is detected that the current faulty module is a chassis, a prompt message is sent to the user indicating that the user needs to replace the chassis, controller or hard disk, so as to repair the chassis; Wherein, the determining of the current fault module according to the distribution of the chassis where each of the target hard disks is located and the target port status of each of the target hard disks includes: When it is detected that the target hard disks are distributed in the same chassis and the target ports of the target hard disks are the same, determining that the current faulty module is a controller connected to the target port; If the number of the target hard disks is less than or equal to a third preset number, when it is detected that the target hard disks are distributed in the same chassis and the target ports of the target hard disks are different, it is determined that the current faulty module is the target hard disk, or that the current faulty module is the controller connected to the target ports; wherein the third preset number is greater than the first preset number; If the number of the target hard disks is greater than the third preset number, and it is detected that the target hard disks are distributed in the same chassis and the target ports of the target hard disks are different, it is determined that the current fault module is the chassis where the target hard disks are located; When it is detected that the target hard disks are distributed in different chassis and the target ports of the target hard disks are the same, determining that the current fault module is a controller connected to the target ports; If the number of the target hard disks is less than or equal to a third preset number, when it is detected that the target hard disks are distributed in different chassis and the target ports of the target hard disks are different, it is determined that the current faulty module is the target hard disk; wherein the third preset number is greater than the first preset number; If the number of target hard disks is greater than the third preset number, when it is detected that the target hard disks are distributed in different chassis and the target ports of the target hard disks are different, it is determined that the current faulty module is the target hard disk, the controller connected to each target port, or the chassis where the target hard disks are located.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the storage system fault handling method described in any one of claims 1 to 11 are implemented.

14. A storage system, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the storage system fault handling method according to any one of claims 1 to 11 when executing the computer program.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the storage system fault handling method according to any one of claims 1 to 11 are implemented.

Images