Fault recovery method and device for shared card in storage cluster

Abstract

The invention discloses a fault recovery method and device for a shared card under a storage cluster, and relates to the technical field of storage clusters, and the method comprises the steps: determining a detection index, and obtaining detection data; inputting the detection data into a pre-constructed fault detection model to obtain a fault detection value of the to-be-detected shared card under the corresponding PCIe channel; judging whether the fault detection values under all PCIe channels meet certain fault detection conditions or not; if all the conditions are met, it is judged that the shared card to be detected breaks down, and resetting is carried out till the first certain health condition is met; and if the first certain health condition is partially met, judging that the channel has a fault, and resetting until a second certain health condition is met. Therefore, the technical problems of high delay, low accuracy, low stability, low reliability and incapability of realizing full-process full-automatic fault detection, isolation and automatic recovery are solved, and the technical effects of quickly detecting faults, performing fault isolation as soon as possible, performing self-recovery and improving reliability and usability are achieved.

Description

Technical Field

[0001] The present application relates to the technical field of storage clusters, and in particular to a fault recovery method and device for a shared card in a storage cluster. Background Art

[0002] In the related technology, the number of packets sent by the network card whose corresponding packet return value is less than or equal to the first set value can be obtained, and based on the number of packets corresponding to the first and Nth fault detections, the detection result of whether the network card has a network card packet sending failure can be obtained; the controller can also send heartbeat packets to the network card periodically. If no response is received for 3 or more consecutive times, the network card is determined to be faulty; the communication status of the network card port can also be used to judge. If the communication status of the network port is abnormal, the network card driver, status register, firmware, and clock signal of the network card to which the network port belongs are subjected to fault detection.

[0003] However, in related technologies, fault detection takes a long time and cannot meet the requirements of rapid detection and rapid processing. It has high lag and poor real-time performance. Fault judgment is performed by sending packets, signals, or network port communication status, which is easily affected by network congestion or instantaneous interference, and is a single indicator detection method with low accuracy. The judgment method is too simple, has certain randomness and uncertainty, and lacks stability. In a multi-controller storage cluster, it can only be detected by one controller, which is not credible enough and has low reliability. Summary of the Invention

[0004] The present application provides a fault recovery method and device for a shared card in a storage cluster to at least solve the technical problems in related technologies, such as high latency, low accuracy, low stability, and low reliability of fault detection, as well as the inability to achieve full-process and fully automated fault detection, fault isolation, automatic recovery, and recovery detection in a multi-controller storage cluster.

[0005] The present application provides a method for recovering a shared card fault in a storage cluster, wherein the storage cluster has at least one shared card and at least two controllers, and each shared card has at least the same number of PCIe (Peripheral Component Interconnect) as the corresponding controller. express, a high-speed serial computer expansion bus interface) channel to communicate with a corresponding controller using the PCIe channel, wherein the method includes the following steps: determining at least one corresponding detection indicator according to a shared card to be tested among the at least one shared card, and obtaining detection data of the shared card to be tested under the corresponding detection indicator based on the at least one detection indicator; inputting the detection data into a pre-built fault detection model to obtain a fault detection value of the shared card to be tested under the corresponding PCIe channel; judging whether the fault detection values ​​under all PCIe channels meet a preset fault detection condition; if the fault detection values ​​under all PCIe channels meet the preset fault detection condition, determining that the shared card to be tested has failed, and resetting the shared card to be tested that has failed until the reset shared card to be tested meets a first preset health condition; if the fault detection values ​​under all PCIe channels partially meet the preset fault detection condition, determining that the PCIe channel that meets the preset fault detection condition has failed, and resetting the PCIe channel that has failed until the reset PCIe channel meets a second preset health condition.

[0006] Optionally, in one embodiment of the present application, before inputting the detection data into a pre-built fault detection model, it also includes: determining at least one detection indicator of the target shared card based on the card information of the target shared card; based on the at least one detection indicator, obtaining the target detection data of the target shared card under the corresponding detection indicator; based on the target detection data, determining the weight information of the corresponding target detection data under different detection indicators; and constructing the fault detection model based on the weight information, the at least one detection indicator and the target detection data.

[0007] Optionally, in one embodiment of the present application, resetting the faulty shared card to be tested until the reset shared card to be tested meets a first preset health condition includes: removing the faulty shared card to be tested from the storage cluster; and resetting the removed shared card to be tested until the reset shared card to be tested meets the first preset health condition.

[0008] Optionally, in one embodiment of the present application, resetting the failed PCIe channel until the reset PCIe channel meets a second preset health condition includes: removing the failed PCIe channel from the shared card to be tested; and resetting the removed PCIe channel until the reset PCIe channel meets the second preset health condition.

[0009] Optionally, in one embodiment of the present application, resetting the removed PCIe channel until the reset PCIe channel meets the second preset health condition includes: obtaining the detection cycle when the removed PCIe channel is reset and repaired; resetting the removed PCIe channel in each detection cycle until the reset PCIe channel meets the second preset health condition.

[0010] Optionally, in one embodiment of the present application, before resetting the failed shared card to be tested, the method further includes: moving data of the failed shared card to be tested to a healthy shared card to be tested, so as to obtain a first data transmission result of the storage cluster.

[0011] Optionally, in one embodiment of the present application, before resetting the failed PCIe channel, the method further includes: moving data of the failed PCIe channel to a healthy PCIe channel to obtain a second data transmission result of the shared card to be tested.

[0012] Optionally, in one embodiment of the present application, the expression of the pre-built fault detection model may be, but is not limited to: , in, is the weight of the controller node detection indicator, It is the detection data of a certain detection indicator of the controller node. Indicates the total number of detection indicators, is the fault detection value corresponding to the PCIe channel, Indicates the number of detection indicators, Indicates the number of times a certain detection indicator is tested. Indicates the detection data of a certain detection indicator of the controller node under a certain detection. Indicates the number of detections for a certain detection indicator.

[0013] The present application also provides a fault recovery device for a shared card in a storage cluster, wherein the storage cluster has at least one shared card and at least two controllers, and each shared card has at least the same number of PCIe channels as the corresponding controller, so as to communicate with the corresponding controller by using the PCIe channels, wherein the device includes: a first acquisition module for determining at least one corresponding detection indicator according to the shared card to be tested in the at least one shared card, and obtaining detection data of the shared card to be tested under the corresponding detection indicator based on the at least one detection indicator; a generation module for inputting the detection data into a pre-built fault detection model to obtain the fault detection value of the shared card to be tested under the corresponding PCIe channel ; A judgment module, used to judge whether the fault detection values ​​under all PCIe channels meet the preset fault detection conditions; a first reset module, used to judge that the shared card to be tested has a fault when the fault detection values ​​under all PCIe channels meet the preset fault detection conditions, and reset the shared card to be tested that has a fault until the reset shared card to be tested meets the first preset health condition; a second reset module, used to judge that the PCIe channel that meets the preset fault detection condition has a fault when the fault detection values ​​under all PCIe channels partially meet the preset fault detection conditions, and reset the PCIe channel that has a fault until the reset PCIe channel meets the second preset health condition.

[0014] Optionally, in one embodiment of the present application, it also includes: a first determination module, used to determine at least one detection indicator of the target shared card based on the card information of the target shared card before inputting the detection data into a pre-built fault detection model; a second acquisition module, used to obtain the target detection data of the target shared card under the corresponding detection indicator based on the at least one detection indicator; a second determination module, used to determine the weight information of the corresponding target detection data under different detection indicators based on the target detection data; a construction module, used to construct the fault detection model based on the weight information, the at least one detection indicator and the target detection data.

[0015] Optionally, in one embodiment of the present application, the first reset module includes: a first removal unit, used to remove the faulty shared card to be tested from the storage cluster; a first reset unit, used to reset the removed shared card to be tested until the reset shared card to be tested meets the first preset health condition.

[0016] Optionally, in one embodiment of the present application, the second reset module includes: a second removal unit, used to remove the faulty PCIe channel from the shared card to be tested; and a third reset unit, used to reset the removed PCIe channel until the reset PCIe channel meets the second preset health condition.

[0017] Optionally, in one embodiment of the present application, the third reset unit includes: an acquisition subunit, used to obtain the detection cycle when the removed PCIe channel is reset and repaired; a reset subunit, used to reset the removed PCIe channel within each detection cycle until the reset PCIe channel meets the second preset health condition.

[0018] Optionally, in one embodiment of the present application, it further includes: a first moving module, used to move the data of the failed shared card under test to a healthy shared card under test before resetting the failed shared card under test, so as to obtain the first data transmission result of the storage cluster.

[0019] Optionally, in one embodiment of the present application, it further includes: a second movement module, used to move the data of the failed PCIe channel to a healthy PCIe channel before resetting the failed PCIe channel to obtain a second data transmission result of the shared card to be tested.

[0020] Optionally, in one embodiment of the present application, the expression of the pre-built fault detection model may be, but is not limited to: , in, is the weight of the controller node detection indicator, It is the detection data of a certain detection indicator of the controller node. Indicates the total number of detection indicators, is the fault detection value corresponding to the PCIe channel, Indicates the number of detection indicators, Indicates the number of times a certain detection indicator is tested. Indicates the detection data of a certain detection indicator of the controller node under a certain detection. Indicates the number of detections for a certain detection indicator.

[0021] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned methods for recovering a shared card in a storage cluster when executing the computer program.

[0022] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for recovering a shared card in a storage cluster are implemented.

[0023] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned methods for recovering a shared card in a storage cluster when executed by a processor.

[0024] The embodiment of the present application can first determine at least one detection indicator corresponding to the shared card to be tested, and then determine the detection data under the corresponding detection indicator, and input the detection data into a pre-built fault detection model to obtain the fault detection value under the PCIe channel, and then determine whether the fault detection values ​​under all PCIe channels meet certain fault detection conditions: if all meet, it is determined that the shared card to be tested has a fault, and the faulty shared card to be tested is reset until the reset shared card to be tested meets the first certain health condition; otherwise, it is determined that the PCIe channel has a fault, and the faulty PCIe channel is reset until the reset PCIe channel meets the second certain health condition. As a result, the high latency, low accuracy, low stability and low reliability of fault detection, as well as the technical problems of inability to achieve full-process and fully automated fault detection, fault isolation, automatic recovery and recovery detection in a multi-control storage cluster are solved, and the storage cluster shared card quickly detects faults, isolates faults as early as possible to avoid worse effects, and can recover by itself to eliminate faults, thereby greatly improving the reliability and availability of the storage cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 A flowchart of a method for recovering a shared card in a storage cluster provided by an embodiment of the present application; Figure 2 A flowchart of the working principle of a fault recovery method for a shared card in a storage cluster provided according to one embodiment of the present application; Figure 3 The present invention is a block diagram of a fault recovery device for a shared card in a storage cluster provided according to an embodiment of the present application.

[0027] Reference numerals: Among them, 10 is a fault recovery device for a shared card in a storage cluster; 100 is a first acquisition module, 200 is a generation module, 300 is a judgment module, 400 is a first reset module, and 500 is a second reset module. DETAILED DESCRIPTION

[0028] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0030] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0031] An embodiment of the present application provides a fault recovery method for a shared card in a storage cluster. The method is described in detail in conjunction with the execution flow of the fault recovery method for a shared card in a storage cluster.

[0032] Specifically, Figure 1 The present invention provides a flowchart of a method for recovering a shared card fault in a storage cluster according to an embodiment of the present application.

[0033] like Figure 1 As shown, the shared card fault recovery method in the storage cluster includes at least one shared card and at least two controllers. Each shared card has at least the same number of PCIe channels as the corresponding controller, so as to communicate with the corresponding controller using the PCIe channels. The method includes the following steps: In step S101, at least one corresponding detection indicator is determined according to a shared card to be tested among at least one shared card, and based on the at least one detection indicator, detection data of the shared card to be tested under the corresponding detection indicator is acquired.

[0034] It is understandable that in a multi-controller storage cluster, a shared smart network card (referred to as a shared card) is a key component used to achieve data sharing and communication between multiple controllers. However, once a shared card fails, it may cause data transmission interruption, performance degradation, or even paralysis of the entire storage cluster. Since the storage cluster has an embedded management service OSES (Organic SAS Enclosure Service), the detection link is extremely short, the resource utilization rate is extremely low, and the transmission efficiency is high. It is possible to set a millisecond-level fault detection cycle without worrying about affecting the performance of the storage cluster. Therefore, the embodiments of the present application can fully utilize and tap into the functional characteristics of OSES, and then use OSES to achieve periodic fault detection of the shared card to be tested.

[0035] Furthermore, in the embodiments of the present application, the detection indicators may include, but are not limited to, data transmission delay, error rate, and bandwidth utilization, and are not specifically limited in this application. Specifically, data transmission delay can be calculated by sending a test data packet and measuring the round-trip time; the error rate can be calculated by counting the ratio of the number of data packets transmitted with errors to the total number of data packets within the detection period; and bandwidth utilization can be calculated by calculating the ratio of the actual amount of data transmitted to the maximum bandwidth of the shared card.

[0036] As a possible implementation method, the embodiment of the present application can first determine the detection indicators of the shared card to be tested, and then obtain the detection data of the shared card to be tested under the corresponding detection indicators.

[0037] Exemplarily, an embodiment of the present application can utilize OSES to implement periodic fault detection of the shared card to be tested. By setting the detection time period T, each controller performs fault detection on the shared card to be tested within each period. The detection indicators may include but are not limited to data transmission delay D, error rate E, and bandwidth utilization U. This application does not impose specific restrictions, and the corresponding data transmission delay data, error rate data, and bandwidth utilization data are obtained.

[0038] The embodiment of the present application can select appropriate detection indicators according to different usage scenarios, and then obtain corresponding detection data, so as to quickly locate the fault position of the shared card to be tested, save detection costs, and improve the availability and reliability of the system.

[0039] Optionally, in one embodiment of the present application, before inputting the detection data into a pre-built fault detection model, the method further includes: determining at least one detection indicator of the target shared card based on the card information of the target shared card; obtaining target detection data of the target shared card under the corresponding detection indicator based on the at least one detection indicator; determining weight information corresponding to the target detection data under different detection indicators based on the target detection data; and constructing a fault detection model based on the weight information, the at least one detection indicator, and the target detection data. The expression of the pre-built fault detection model may be, but is not limited to, the following: , in, is the weight of the controller node detection indicator, It is the detection data of a certain detection indicator of the controller node. Indicates the total number of detection indicators, is the fault detection value corresponding to the PCIe channel, Indicates the number of detection indicators, Indicates the number of times a certain detection indicator is tested. Indicates the detection data of a certain detection indicator of the controller node under a certain detection. Indicates the number of detections for a certain detection indicator.

[0040] It is understandable that the embodiment of the present application assumes that a controller node uses a target shared card, and the detection indicators may include but are not limited to (i.e., the data transmission delay D, error rate E, and bandwidth utilization U mentioned above, which are not specifically limited in this application), the target detection data of the controller node is represented by vector In other words, further, the embodiment of the present application can determine the weight information of the target detection data corresponding to different detection indicators according to the target detection data, for example, It can be set to 0.4. and Both can be set to 0.3, and can be specifically set by technicians in this field according to actual conditions. This application does not impose any specific restrictions.

[0041] Furthermore, the embodiment of the present application can construct a fault detection model based on weight information, detection indicators and target detection data. The expression of the pre-constructed fault detection model can be, but is not limited to: , in, The weight of the controller node detection index can be dynamically adjusted according to the importance of the detection index and the historical failure rate. The sum of is 1, when hour, It can be understood as the weight corresponding to the data transmission delay data, or it can be expressed as ,when hour, It can be understood as the weight corresponding to the error rate data, or it can be expressed as ,when hour, It can be understood as the weight corresponding to the bandwidth utilization data, or it can be expressed as , It is the detection data of a certain detection indicator of the controller node. Indicates the total number of detection indicators, is the fault detection value corresponding to the PCIe channel, Indicates the number of detection indicators, Indicates the number of times a certain detection indicator is tested. Indicates the detection data of a certain detection indicator of the controller node under a certain detection. Indicates the number of detections for a certain detection indicator.

[0042] The embodiment of the present application can construct a fault detection model based on the detection indicators, the target detection data under the corresponding detection indicators, and the weight information of the corresponding target detection data. The detection accuracy and adaptability of the model can be improved by customizing the detection indicators. By obtaining the target detection data, irrelevant data can be prevented from affecting the accuracy of the model, the model training efficiency can be improved, and key indicators can be optimized through weight distribution to improve the accuracy of diagnosis.

[0043] In step S102 , the detection data is input into a pre-built fault detection model to obtain a fault detection value of the shared card to be tested in the corresponding PCIe channel.

[0044] It can be understood that each shared card in the embodiment of the present application has at least the same number of PCIe channels as the corresponding controller. Therefore, the embodiment of the present application can input the detection data into a pre-built fault detection model to obtain the fault detection values ​​under different PCIe channels.

[0045] Among them, when the detection indicator is data transmission delay, the fault detection value is only the fault detection value of the data transmission delay data; when the detection indicators are data transmission delay and bandwidth utilization, the fault detection value is the fault detection value corresponding to the data transmission delay data and bandwidth utilization data; when the detection indicators are data transmission delay, error rate and bandwidth utilization, the fault detection value is the fault detection value corresponding to the data transmission delay data, error rate data and bandwidth utilization data.

[0046] For example, in some embodiments, when the detection indicators of the embodiment of the present application are data transmission delay, error rate and bandwidth utilization, Set to 0.4, and The data transmission delay data, error rate data and bandwidth utilization data are input into the pre-built fault detection model to obtain the fault detection values ​​under different PCIe channels. The fault detection value includes the fault detection value corresponding to the data transmission delay data, the error rate data, and the bandwidth utilization data.

[0047] In some embodiments, when the detection indicators of the embodiment of the present application are data transmission delay and bandwidth utilization, Set to 0.3, Set it to 0.7, and then get the pre-built fault detection model, and input the data transmission delay data and bandwidth utilization data into the pre-built fault detection model to get the fault detection value under different PCIe channels. The fault detection value includes the fault detection value corresponding to the data transmission delay data and the bandwidth utilization data.

[0048] In some embodiments, when the detection indicator is data transmission delay, the Set it to 1 to obtain the pre-built fault detection model, and input the data transmission delay data into the pre-built fault detection model to obtain the fault detection value under different PCIe channels. The fault detection value includes a fault detection value corresponding to the data transmission delay data.

[0049] The embodiments of the present application can use a pre-built fault detection model to obtain fault detection values ​​corresponding to different PCIe channels, realize automatic model detection without manual intervention, reduce maintenance costs, and improve the real-time and efficiency of model detection.

[0050] In step S103 , it is determined whether the fault detection values ​​of all PCIe channels meet a preset fault detection condition.

[0051] It can be understood that certain fault conditions in the embodiments of the present application may include but are not limited to data transmission delay fault conditions, error rate fault conditions, bandwidth utilization fault conditions, and combinations of different fault conditions, etc., which can be specifically set by technicians in this field according to actual conditions. This application does not impose specific restrictions and uses combined fault conditions to locate defects and improve fault location accuracy.

[0052] During actual execution, the OSES of each controller in the embodiment of the present application can use a pre-built fault detection model to determine whether the fault detection values ​​under all PCIe channels meet certain fault detection conditions.

[0053] Exemplarily, in some embodiments, the embodiments of the present application can determine whether the fault detection value including data transmission delay data, error rate data and bandwidth utilization data meets the combined conditions of data transmission delay fault condition, error rate fault condition and bandwidth utilization fault condition.

[0054] In some embodiments, the embodiments of the present application may determine whether a fault detection value including data transmission delay data and bandwidth utilization data satisfies a combination condition of a data transmission delay fault condition and a bandwidth utilization fault condition.

[0055] Optionally, in one embodiment of the present application, before resetting the failed shared card to be tested, the method further includes: moving data of the failed shared card to be tested to a healthy shared card to be tested, so as to obtain a first data transmission result of the storage cluster.

[0056] In some embodiments, when it is determined that a shared card to be tested fails, the embodiments of the present application can move the corresponding data to a healthy shared card to be tested, thereby obtaining a first data transmission result of the storage cluster.

[0057] Illustratively, once it is confirmed that the shared card under test has failed, the OSES may move the data corresponding to the failed shared card under test to a healthy shared card under test, thereby obtaining a first data transmission result of the storage cluster.

[0058] The embodiment of the present application can move the data of the faulty shared card under test to a healthy shared card under test, and then perform data transmission, ensuring that when the shared card under test fails, data transmission is not affected or data is not lost, thereby improving the availability and reliability of the system.

[0059] In step S104, if the fault detection values ​​under all PCIe channels meet the preset fault detection condition, it is determined that the shared card under test has failed, and the failed shared card under test is reset until the reset shared card under test meets the first preset health condition.

[0060] In some embodiments, embodiments of the present application can determine that a shared card under test has failed if the fault detection values ​​under all PCIe channels meet certain fault detection conditions, and reset the failed shared card under test until the reset shared card under test meets a first certain health condition. The first certain health condition can be set by those skilled in the art based on actual conditions and is not specifically limited by this application.

[0061] Illustratively, in an embodiment of the present application, when the fault detection values ​​under all PCIe channels meet certain fault detection conditions, it is determined that a fault has occurred on the shared card to be tested, and OSES immediately adopts an automatic recovery mechanism: the faulty shared card to be tested is powered on and off, that is, the faulty shared card to be tested is reset, and after the reset, OSES performs repair detection until the reset shared card to be tested meets a first certain health condition, for example, the fault detection values ​​of the reset shared card to be tested do not meet certain fault detection conditions.

[0062] Optionally, in one embodiment of the present application, the faulty shared card to be tested is reset until the reset shared card to be tested meets the first preset health condition, including: removing the faulty shared card to be tested from the storage cluster; and resetting the removed shared card to be tested until the reset shared card to be tested meets the first preset health condition.

[0063] It is understandable that when resetting a faulty shared card under test, the embodiment of the present application may first temporarily remove the faulty shared card under test from the storage cluster, and then reset the removed shared card under test until the first certain health condition is met.

[0064] For example, when a fault occurs in the shared card to be tested in the embodiment of the present application, OSES immediately starts the isolation mechanism and temporarily removes the faulty shared card to be tested from the storage cluster. After isolation, OSES immediately adopts an automatic recovery mechanism: powering on and off the faulty shared card to be tested, that is, resetting the faulty shared card to be tested. After the reset, OSES performs repair detection until the reset shared card to be tested meets the first certain health condition.

[0065] The embodiment of the present application can remove the faulty shared card under test from the storage cluster, and then reset the removed shared card under test until the first certain health condition is met. This is fully automated and does not require manual intervention, ensuring that the shared card can quickly restore its service capabilities in the event of a failure, thereby improving the reliability of the system.

[0066] Optionally, in one embodiment of the present application, before resetting the failed PCIe channel, the method further includes: moving data of the failed PCIe channel to a healthy PCIe channel to obtain a second data transmission result of the shared card to be tested.

[0067] In some embodiments, when it is determined that a PCIe channel fails, the embodiments of the present application can move the corresponding data to a healthy PCIe channel, thereby obtaining a second data transmission result of the shared card under test.

[0068] Illustratively, once the embodiment of the present application confirms that a PCIe channel has failed, OSES only retains the healthy PCIe channel for data transmission, and uses a data redirection algorithm to distribute the data of the failed PCIe channel to other healthy PCIe channels, that is, to move the data corresponding to the failed PCIe channel to the healthy PCIe channel, thereby obtaining the second data transmission result of the shared card to be tested.

[0069] Among them, the embodiment of the present application can define a healthy PCIe channel set as ,in, The minimum detection index threshold for a healthy PCIe channel can be set by technicians in this field according to actual conditions, and this application does not impose any specific restrictions.

[0070] The embodiment of the present application can move data from a failed PCIe channel to a healthy PCIe channel for data transmission, ensuring that data transmission is not affected or data is not lost when a PCIe channel fails, thereby improving the availability and reliability of the system.

[0071] In step S105, if the fault detection values ​​under all PCIe channels partially meet the preset fault detection conditions, it is determined that the PCIe channels that meet the preset fault detection conditions have failed, and the failed PCIe channels are reset until the reset PCIe channels meet the second preset health condition.

[0072] In some embodiments, when the fault detection values ​​of all PCIe channels partially meet certain fault detection conditions, the PCIe channels that meet the certain fault detection conditions are determined to be faulty, and the faulty PCIe channels are reset until the reset PCIe channels meet a second certain health condition. The second certain health condition can be set by those skilled in the art based on actual circumstances and is not specifically limited in this application.

[0073] Illustratively, an embodiment of the present application can determine that PCIe channel A and PCIe channel B have failed when the fault detection values ​​of PCIe channel A and PCIe channel B meet certain fault detection conditions, and reset PCIe channel A and PCIe channel B until the reset PCIe channel A and PCIe channel B meet a second certain health condition.

[0074] Optionally, in one embodiment of the present application, the faulty PCIe channel is reset until the reset PCIe channel meets a second preset health condition, including: removing the faulty PCIe channel from the shared card to be tested; and resetting the removed PCIe channel until the reset PCIe channel meets the second preset health condition.

[0075] In some embodiments, after determining that a PCIe channel fails, the embodiments of the present application may remove the failed PCIe channel from the shared card to be tested, and then reset the removed PCIe channel until a second certain health condition is met.

[0076] For example, in the embodiment of the present application, after determining that a PCIe channel fails, the failed PCIe channel is located as the failed channel. , OSES immediately starts the isolation mechanism, removes the failed PCIe channel from the shared card under test, and immediately adopts the automatic recovery mechanism, that is, the failed PCIe channel between the controller node and the shared card under test Reset until the detection index of the reset PCIe channel meets or exceeds , it is determined that the reset PCIe channel meets the second certain health condition, that is, the reset PCIe channel has been repaired and can be used again.

[0077] The embodiment of the present application can remove the faulty PCIe channel from the shared card to be tested and reset it until a second certain health condition is met. This is fully automated and requires no human intervention, ensuring that the shared card can quickly restore its service capabilities in the event of a failure, thereby improving the reliability of the system.

[0078] Optionally, in one embodiment of the present application, the removed PCIe channel is reset until the reset PCIe channel meets a second preset health condition, including: obtaining the detection cycle when the removed PCIe channel is reset and repaired; resetting the removed PCIe channel in each detection cycle until the reset PCIe channel meets the second preset health condition.

[0079] It is understandable that the embodiment of the present application can use OSES to perform periodic repair detection on the removed PCIe channel, that is, OSES will reset the removed PCIe channel, and after the reset, OSES will perform repair detection. If the detection index of the removed PCIe channel is greater than , it is determined that the reset PCIe channel meets the second certain health condition and can be used again.

[0080] In some embodiments, the embodiments of the present application first obtain the detection cycle when the removed PCIe channel is reset and repaired, and reset the removed PCIe channel in each detection cycle until the reset PCIe channel meets the second certain health condition.

[0081] For example, the embodiment of the present application can use OSES to perform periodic repair detection on the removed PCIe channel, and the detection period is , and adopt an automatic recovery mechanism in each detection cycle, that is, to recover the PCIe channel that fails between the controller node and the shared card to be tested Perform a reset.

[0082] Furthermore, in the embodiment of the present application, if the detection index of the removed PCIe channel is Greater than , it is determined that the reset PCIe channel meets the second certain health condition and can be used again.

[0083] The embodiment of the present application can obtain the detection cycle when the removed PCIe channel is reset and repaired, and reset the removed PCIe channel within each detection cycle until the reset PCIe channel meets the second certain health condition. Through periodic detection, potential problems can be discovered in time, fault recovery can be performed quickly, fault accumulation can be avoided, and long-term stability can be guaranteed.

[0084] In order to fully automate the fault recovery method of shared cards in storage clusters, the following methods are designed in combination with the functional features of OSES: Figure 2 The fully automated process shown.

[0085] in, Figure 2 The present invention is a flowchart illustrating the working principle of a fault recovery method for a shared card in a storage cluster provided according to an embodiment of the present application.

[0086] Step S201: Initialize the storage cluster.

[0087] Step S202: Actively deploy a Docker container through OSES.

[0088] In this embodiment of the present application, OSES can actively deploy Docker containers and perform cluster management to obtain detection indicators of the shared card to be tested.

[0089] Step S203: obtaining detection indicators of the shared card to be tested through the Docker container.

[0090] Among them, the embodiment of the present application can realize periodic fault detection of the shared card to be tested through OSES, and obtain the detection indicators of the shared card to be tested in each detection cycle. The main contents may be: preparing the Docker environment; preparing the data acquisition form; writing the Dockerfile; building the Docker image; running the Docker container; obtaining the detection indicators; and saving the data.

[0091] Step S204: Perform periodic fault detection on the shared card to be tested through OSES.

[0092] Step S205: Acquire fault detection values ​​corresponding to different PCIe channels in combination with a pre-built fault detection model.

[0093] Step S206: Determine whether all fault detection values ​​meet certain fault detection conditions.

[0094] In this embodiment of the present application, if the fault detection values ​​under all PCIe channels meet certain fault detection conditions, step S207 is executed; otherwise, step S210 is executed.

[0095] Step S207: determining whether the shared card to be tested fails.

[0096] Step S208: Activate the isolation mechanism of the shared card to be tested.

[0097] Among them, after the OSES embodiment of the present application immediately starts the isolation mechanism of the shared card under test, the faulty shared card under test is temporarily removed from the storage cluster, and the data corresponding to the faulty shared card under test is moved to the healthy shared card under test, thereby obtaining the first data transmission result of the storage cluster.

[0098] Step S209: Activate the automatic recovery mechanism of the shared card to be tested.

[0099] Among them, in the embodiment of the present application, OSES immediately starts the automatic recovery mechanism of the shared card to be tested, and then powers on and off the faulty shared card to be tested, that is, resets the faulty shared card to be tested. After the reset, OSES performs repair detection until the reset shared card to be tested meets the first certain health condition.

[0100] Step S210: Determine whether a PCIe channel failure occurs.

[0101] Step S211: Start the PCIe channel isolation mechanism.

[0102] Among them, after the OSES embodiment of the present application immediately starts the PCIe channel isolation mechanism, the faulty PCIe channel is removed from the shared card to be tested, and only the healthy PCIe channel is retained for data transmission, and a data redirection algorithm is used to distribute the data of the faulty PCIe channel to other healthy PCIe channels, that is, the data corresponding to the faulty PCIe channel is moved to the healthy PCIe channel, thereby obtaining the second data transmission result of the shared card to be tested.

[0103] Step S212: Start the PCIe channel automatic recovery mechanism.

[0104] Among them, after the embodiment of the present application OSES immediately starts the PCIe channel automatic recovery mechanism, it performs periodic repair detection on the removed PCIe channel, and in each detection cycle, it detects the PCIe channel that fails between the controller node and the shared card to be tested. Reset, if the detection indicator of the removed PCIe channel is Greater than , it is determined that the reset PCIe channel meets the second certain health condition and can be used again.

[0105] Step S213: OSES reports the fault.

[0106] In this embodiment of the present application, OSES can proactively report the fault detection, isolation, and repair results of the shared card in the storage cluster in the form of logs.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0108] According to the fault recovery method of a shared card in a storage cluster proposed in an embodiment of the present application, at least one detection indicator corresponding to the shared card to be tested can be determined first, and then the detection data under the corresponding detection indicator can be determined, and the detection data can be input into a pre-built fault detection model to obtain a fault detection value under the PCIe channel, and then it can be determined whether the fault detection values ​​under all PCIe channels meet certain fault detection conditions: if all of them meet, it is determined that the shared card to be tested has failed, and the failed shared card to be tested is reset until the reset shared card to be tested meets the first certain health condition; otherwise, it is determined that the PCIe channel has failed, and the failed PCIe channel is reset until the reset PCIe channel meets the second certain health condition. As a result, the high latency, low accuracy, low stability and low reliability of fault detection, as well as the technical problems of inability to achieve full-process and fully automated fault detection, fault isolation, automatic recovery and recovery detection in a multi-control storage cluster are solved, and the storage cluster shared card can quickly detect faults, isolate faults as early as possible to avoid worse effects, and can recover by itself to eliminate faults, thereby greatly improving the reliability and availability of the storage cluster.

[0109] An embodiment of the present application also provides a fault recovery device for a shared card in a storage cluster.

[0110] Figure 3 The present invention is a block diagram of a fault recovery device for a shared card in a storage cluster provided according to an embodiment of the present application.

[0111] like Figure 3 As shown, the fault recovery device 10 of the shared card in the storage cluster has at least one shared card and at least two controllers in the storage cluster, and each shared card has at least the same number of PCIe channels as the corresponding controller, so as to communicate with the corresponding controller by using the PCIe channels, wherein the fault recovery device of the shared card in the storage cluster includes: a first acquisition module 100, a generation module 200, a judgment module 300, a first reset module 400 and a second reset module 500.

[0112] The first acquisition module 100 is configured to determine at least one corresponding detection indicator according to the shared card to be tested among the at least one shared card, and acquire detection data of the shared card to be tested under the corresponding detection indicator based on the at least one detection indicator.

[0113] The generating module 200 is used to input the detection data into a pre-built fault detection model to obtain a fault detection value of the shared card to be tested in the corresponding PCIe channel.

[0114] The judgment module 300 is used to judge whether the fault detection values ​​of all PCIe channels meet the preset fault detection conditions.

[0115] The first reset module 400 is used to determine that the shared card under test has failed when the fault detection values ​​under all PCIe channels meet the preset fault detection conditions, and reset the failed shared card under test until the reset shared card under test meets the first preset health condition.

[0116] The second reset module 500 is used to determine that a PCIe channel that meets the preset fault detection condition has a fault when the fault detection values ​​under all PCIe channels partially meet the preset fault detection condition, and reset the faulty PCIe channel until the reset PCIe channel meets the second preset health condition.

[0117] Optionally, in one embodiment of the present application, it further includes: a first determination module, a second acquisition module, a second determination module and a construction module.

[0118] The first determination module is configured to determine at least one detection index of the target shared card according to the card information of the target shared card before inputting the detection data into the pre-built fault detection model.

[0119] The second acquisition module is used to acquire target detection data of the target shared card under the corresponding detection indicator based on at least one detection indicator.

[0120] The second determination module is used to determine weight information corresponding to the target detection data under different detection indicators based on the target detection data.

[0121] A construction module is used to construct a fault detection model based on weight information, at least one detection indicator and target detection data.

[0122] Optionally, in one embodiment of the present application, the first resetting module 400 includes: a first removing unit and a first resetting unit.

[0123] The first removing unit is configured to remove a faulty shared card to be tested from the storage cluster.

[0124] The first resetting unit is configured to reset the removed shared card to be tested until the reset shared card to be tested meets a first preset health condition.

[0125] Optionally, in one embodiment of the present application, the second resetting module 500 includes: a second removing unit and a third resetting unit.

[0126] The second removing unit is configured to remove the failed PCIe channel from the shared card to be tested.

[0127] The third resetting unit is configured to reset the removed PCIe channel until the reset PCIe channel meets a second preset health condition.

[0128] Optionally, in one embodiment of the present application, the third reset unit includes: an acquisition subunit and a reset subunit.

[0129] The acquisition subunit is used to obtain the detection cycle of the removed PCIe channel when it is reset and repaired.

[0130] The reset subunit is used to reset the removed PCIe channel in each detection cycle until the reset PCIe channel meets the second preset health condition.

[0131] Optionally, in one embodiment of the present application, it further includes: a first moving module.

[0132] The first moving module is used to move data of the failed shared card to be tested to a healthy shared card to be tested before resetting the failed shared card to obtain a first data transmission result of the storage cluster.

[0133] Optionally, in one embodiment of the present application, it further includes: a second mobile module.

[0134] The second moving module is used to move data of the failed PCIe channel to a healthy PCIe channel before resetting the failed PCIe channel, so as to obtain a second data transmission result of the shared card to be tested.

[0135] Optionally, in one embodiment of the present application, the expression of the pre-built fault detection model may be, but is not limited to: , in, is the weight of the controller node detection indicator, It is the detection data of a certain detection indicator of the controller node. Indicates the total number of detection indicators, is the fault detection value corresponding to the PCIe channel, Indicates the number of detection indicators, Indicates the number of times a certain detection indicator is tested. Indicates the detection data of a certain detection indicator of the controller node under a certain detection. Indicates the number of detections for a certain detection indicator.

[0136] For descriptions of features in the embodiments corresponding to the device for recovering a shared card in a storage cluster, please refer to the descriptions of the embodiments corresponding to the method for recovering a shared card in a storage cluster, which will not be described in detail here.

[0137] According to the fault recovery device for a shared card in a storage cluster proposed in an embodiment of the present application, it is possible to first determine at least one detection indicator corresponding to the shared card to be tested, and then determine the detection data under the corresponding detection indicator, and input the detection data into a pre-built fault detection model to obtain a fault detection value under the PCIe channel, and then determine whether the fault detection values ​​under all PCIe channels meet certain fault detection conditions: if all meet, it is determined that the shared card to be tested has failed, and the failed shared card to be tested is reset until the reset shared card to be tested meets the first certain health condition; otherwise, it is determined that the PCIe channel has failed, and the failed PCIe channel is reset until the reset PCIe channel meets the second certain health condition. As a result, the high latency, low accuracy, low stability and low reliability of fault detection, as well as the technical problems of inability to achieve full-process and fully automated fault detection, fault isolation, automatic recovery and recovery detection in a multi-control storage cluster are solved, and the storage cluster shared card quickly detects faults, isolates faults as early as possible to avoid worse effects, and can recover by itself to eliminate faults, thereby greatly improving the reliability and availability of the storage cluster.

[0138] An embodiment of the present application further provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned embodiments of the method for fault recovery of a shared card in a storage cluster.

[0139] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps of any of the above-mentioned embodiments of the method for fault recovery of a shared card in a storage cluster when running.

[0140] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0141] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned embodiments of the method for fault recovery of a shared card in a storage cluster are implemented.

[0142] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned embodiments of the shared card fault recovery method in the storage cluster.

[0143] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0144] The above is a detailed introduction to the fault recovery method for a shared card in a storage cluster provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for recovering a shared card fault in a storage cluster, characterized in that: The storage cluster has at least one shared card and at least two controllers, each shared card has at least the same number of high-speed serial computer expansion bus (PCIe) channels as the number of corresponding controllers, so as to communicate with the corresponding controllers using the PCIe channels. The method includes the following steps: determining at least one corresponding detection indicator according to the shared card to be tested among the at least one shared card, and acquiring detection data of the shared card to be tested under the corresponding detection indicator based on the at least one detection indicator; Inputting the detection data into a pre-built fault detection model to obtain a fault detection value of the shared card to be tested in the corresponding PCIe channel; Determine whether the fault detection values ​​under all PCIe channels meet the preset fault detection conditions; If the fault detection values ​​under all the PCIe channels meet the preset fault detection condition, it is determined that the shared card to be tested has a fault, and the shared card to be tested that has a fault is reset until the reset shared card to be tested meets the first preset health condition; If the fault detection values ​​under all the PCIe channels partially meet the preset fault detection conditions, it is determined that the PCIe channels that meet the preset fault detection conditions have failed, and the failed PCIe channels are reset until the reset PCIe channels meet the second preset health condition.

2. The method according to claim 1, characterized in that Before inputting the detection data into a pre-built fault detection model, the method further includes: determining at least one detection indicator of the target shared card according to the card information of the target shared card; Based on the at least one detection indicator, obtaining target detection data of the target shared card under the corresponding detection indicator; Based on the target detection data, determining weight information corresponding to the target detection data under different detection indicators; The fault detection model is constructed based on the weight information, the at least one detection indicator and the target detection data.

3. The method according to claim 1, characterized in that The step of resetting the faulty shared card to be tested until the reset shared card to be tested meets a first preset health condition includes: Removing the faulty shared card to be tested from the storage cluster; The removed shared card to be tested is reset until the reset shared card to be tested meets the first preset health condition.

4. The method according to claim 1, wherein The resetting of the failed PCIe channel until the reset PCIe channel meets a second preset health condition includes: Removing the failed PCIe channel from the shared card to be tested; The removed PCIe channel is reset until the reset PCIe channel meets the second preset health condition.

5. The method according to claim 4, characterized in that The step of resetting the removed PCIe channel until the reset PCIe channel meets the second preset health condition includes: Obtaining a detection period for resetting and repairing the removed PCIe channel; The removed PCIe channel is reset in each detection cycle until the reset PCIe channel meets the second preset health condition.

6. The method according to claim 1, characterized in that Before resetting the faulty shared card under test, the following steps are also required: The data of the failed shared card to be tested is moved to a healthy shared card to be tested, so as to obtain a first data transmission result of the storage cluster.

7. The method according to claim 1, characterized in that Before resetting the failed PCIe channel, the following steps are also included: The data of the failed PCIe channel is moved to a healthy PCIe channel to obtain a second data transmission result of the shared card to be tested.

8. The method according to claim 1, characterized in that The expression of the pre-built fault detection model is: ， in, is the weight of the controller node detection indicator, It is the detection data of a certain detection indicator of the controller node. Indicates the total number of detection indicators, is the fault detection value corresponding to the PCIe channel, Indicates the number of detection indicators, Indicates the number of times a certain detection indicator is tested. Indicates the detection data of a certain detection indicator of the controller node under a certain detection. Indicates the number of detections for a certain detection indicator.

9. A fault recovery device for a shared card in a storage cluster, characterized in that: The storage cluster has at least one shared card and at least two controllers, each shared card has at least the same number of PCIe channels as the corresponding controller, so as to communicate with the corresponding controller using the PCIe channels, wherein the device includes: a first acquisition module, configured to determine at least one corresponding detection indicator according to the shared card to be tested among the at least one shared card, and acquire, based on the at least one detection indicator, detection data of the shared card to be tested under the corresponding detection indicator; A generating module, configured to input the detection data into a pre-built fault detection model to obtain a fault detection value of the shared card to be tested in the corresponding PCIe channel; A judgment module is used to judge whether the fault detection values ​​under all PCIe channels meet the preset fault detection conditions; a first resetting module, configured to determine that the shared card to be tested has a fault when the fault detection values ​​under all the PCIe channels meet the preset fault detection condition, and reset the shared card to be tested that has a fault until the reset shared card to be tested meets a first preset health condition; The second reset module is configured to determine that a PCIe channel that meets the preset fault detection condition has failed when a portion of the fault detection values ​​under all the PCIe channels meets the preset fault detection condition, and reset the failed PCIe channel until the reset PCIe channel meets the second preset health condition.

10. The device according to claim 9, characterized in that Also includes: a first determining module, configured to determine at least one detection index of the target shared card according to the card information of the target shared card before inputting the detection data into a pre-built fault detection model; A second acquisition module is configured to acquire target detection data of the target shared card under a corresponding detection indicator based on the at least one detection indicator; A second determination module is used to determine weight information corresponding to the target detection data under different detection indicators based on the target detection data; A construction module is used to construct the fault detection model based on the weight information, the at least one detection indicator and the target detection data.

11. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the fault recovery method for a shared card in a storage cluster according to any one of claims 1 to 8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the fault recovery method for a shared card in a storage cluster according to any one of claims 1 to 8.

Images