Method and apparatus for failure recovery of all-flash storage system

Abstract

The application discloses a fault repair method and device of a full-flash storage system, and relates to the technical field of storage system self-repairing, and comprises the following steps: establishing an automatic repair framework, and classifying and processing faults of the full-flash storage system; performing an alarm reporting, identification diagnosis and repair processing flow on each type of fault, and importing detection, diagnosis, repair and feedback operations of each fault into the automatic repair framework, so that automatic identification, detection, repair and feedback can be performed after the fault occurs; and the system decides the next action according to the feedback result, thereby solving the technical problem that, in the related art, the time cost of fault detection and repair of the full-flash storage system is relatively large, and it is difficult to meet the timeliness requirement of customers, achieving the technical effects that automatic identification, detection, repair and feedback can be performed after the fault occurs, so that the fault repair time can be greatly shortened, business interruption caused by the fault or interruption time exceeding the business requirement can be avoided, and the use experience of customers can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of storage system self-repair, and in particular to a fault repair method and device for a full-flash storage system. BACKGROUND

[0002] Due to the complexity of a full-flash storage system, once a fault occurs, it is easy to cause a serious impact on business, and even lead to business downtime. Therefore, repair timeliness is a core index that needs to be considered. However, the repair timeliness is restricted by many factors such as customer business feedback and log collection efficiency, so that the uncertainty of on-site fault repair timeliness is large, and it is difficult to meet the requirements of sensitive customers on timeliness.

[0003] At present, when metadata inspection abnormity occurs in a storage product, the related technology can directly trigger an alarm and stop the service, causing the upper-layer business to be interrupted. Subsequently, maintenance personnel and R&D personnel need to be involved to complete operations such as log collection, fault positioning and repair strategy execution. However, the repair time of such a fault is too long, and it is unable to meet the requirements of customers such as finance who have very high requirements on business repair timeliness.

[0004] In summary, when a fault occurs in a full-flash storage system, the time cost of fault detection and repair by the related technology is large, and it is difficult to meet the requirements of customers on timeliness, which needs to be solved urgently. SUMMARY

[0005] The present application provides a fault repair method and device for a full-flash storage system, to at least solve the technical problem in the related art that the time cost of fault detection and repair for a full-flash storage system is large, and it is difficult to meet the requirements of customers on timeliness.

[0006] The present application provides a fault repair method for a full-flash storage system, comprising the following steps: detecting whether a fault occurs in a full-flash storage system during operation, and determining a fault type of the fault if it is detected that the full-flash storage system has a fault, so as to call a fault diagnosis mechanism corresponding to the fault type from a pre-constructed fault automatic repair model according to the fault type; diagnosing a fault cause of the full-flash storage system by using the fault diagnosis mechanism, and judging whether the full-flash storage system meets a preset repair condition according to the fault cause; if the preset repair condition is met, inputting the fault cause into the fault automatic repair model to call a fault repair mechanism corresponding to the fault cause, and performing a repair operation on the full-flash storage system by using the fault repair mechanism, otherwise, performing a manual repair prompt.

[0007] The application further provides a fault repair device of an all-flash storage system, comprising: a fault detection module, configured to detect whether a fault occurs in the all-flash storage system during operation, and determine a fault type of the fault if it is detected that the fault occurs in the all-flash storage system, so as to call a fault diagnosis mechanism corresponding to the fault type from a pre-constructed fault automatic repair model according to the fault type; a fault diagnosis module, configured to diagnose a fault cause of the all-flash storage system by using the fault diagnosis mechanism, and determine whether the all-flash storage system meets a preset repair condition according to the fault cause; and a fault repair module, configured to input the fault cause into the fault automatic repair model to call a fault repair mechanism corresponding to the fault cause if the preset repair condition is met, and perform a repair operation on the all-flash storage system by using the fault repair mechanism, or otherwise, perform an artificial repair prompt.

[0008] The application further provides an electronic device, comprising: a memory configured to store a computer program; and a processor configured to implement the steps of any of the fault repair methods of the all-flash storage system when executing the computer program.

[0009] The application further provides a non-volatile computer readable storage medium, wherein the non-volatile computer readable storage medium stores a computer program, and the computer program is configured to implement the steps of any of the fault repair methods of the all-flash storage system when executed by a processor.

[0010] The application further provides a computer program product, comprising a computer program, and the computer program is configured to implement the steps of any of the fault repair methods of the all-flash storage system when executed by a processor.

[0011] By the application, whether a fault occurs in the all-flash storage system during operation can be detected, and a fault type of the fault can be determined if it is detected that the fault occurs in the all-flash storage system, so as to call a fault diagnosis mechanism corresponding to the fault type from a pre-constructed fault automatic repair model according to the fault type; a fault cause of the all-flash storage system can be diagnosed by using the fault diagnosis mechanism, and whether the all-flash storage system meets a preset repair condition can be determined according to the fault cause; if the preset repair condition is met, the fault cause is input into the fault automatic repair model to call a fault repair mechanism corresponding to the fault cause, and a repair operation is performed on the all-flash storage system by using the fault repair mechanism, or otherwise, an artificial repair prompt is performed, so that the technical problem that a time cost is large for fault detection and repair of the all-flash storage system in the related art and it is difficult to meet the timeliness requirement of customers can be solved, and the technical effect that automatic identification, detection, repair and feedback are performed after the fault occurs is achieved, so that the time for fault repair can be greatly shortened, business interruption or interruption time caused by the fault is avoided, and the use experience of customers is improved. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating a fault repair method for an all-flash storage system according to an embodiment of this application;

[0014] Figure 2 A schematic diagram of the execution logic for full-cycle processing of simplified pool faults is provided as an embodiment of this application;

[0015] Figure 3 A schematic diagram of the execution logic for log volume fault diagnosis in an all-flash storage system provided as an embodiment of this application;

[0016] Figure 4 This is an example diagram of a fault repair apparatus for an all-flash storage system according to an embodiment of this application.

[0017] Among them, 10 is a fault repair device for all-flash storage systems, 100 is a fault detection module, 200 is a fault diagnosis module, and 300 is a fault repair module. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0019] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0020] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] The specific application environment architecture or specific hardware architecture on which the fault repair method of the combined all-flash storage system depends is described here.

[0022] The embodiments of this application provide a fault repair method for an all-flash storage system.

[0023] like Figure 1 The diagram shown is a flowchart of a fault repair method for an all-flash storage system according to an embodiment of this application. The fault repair method for the all-flash storage system includes the following steps:

[0024] In step S101, it is detected whether a fault occurs in the all-flash storage system during operation, and if a fault is detected in the all-flash storage system, the fault type is determined, so as to call the fault diagnosis mechanism corresponding to the fault type from the pre-built automatic fault repair model according to the fault type.

[0025] Those skilled in the art should understand that in recent years, all-flash array (AFA) storage systems have achieved remarkable results in performance, reliability, intelligence, and domestic production, and have become the mainstream solution in the enterprise storage market. However, all-flash storage systems differ significantly from traditional hard disk drives (HDDs). Their optimization must revolve around the physical characteristics of NAND flash memory, such as dealing with write amplification, extending erase lifetime, and optimizing parallel access. This makes their data organization more complex, covering key aspects such as the mapping management of logical addresses and physical addresses, deduplication metadata management, and garbage collection.

[0026] In the pursuit of high performance and high density, the complexity of data organization in all-flash storage systems has increased significantly, posing new challenges to core indicators such as data security and system reliability. Since storage systems typically support critical business operations, they have extremely high requirements for business recovery capabilities and timeliness, especially in industries like finance and healthcare, where system downtime exceeding half an hour can have a fatal impact on customers. Therefore, improving storage reliability to ensure business continuity is a core focus of the development of all-flash storage systems.

[0027] Therefore, the embodiments of this application can use an automatic detection and repair method for all-flash storage systems to summarize and proactively detect, diagnose, and repair potential faults at the system level, replacing the traditional passive fault handling strategy.

[0028] In actual execution, the embodiment of the present application can monitor the running state of the all-flash storage system in real time. If a fault is detected in the system, the specific type of the fault is further determined, and the fault diagnosis mechanism matched with the fault type is retrieved from the pre-constructed fault automatic repair model according to the determined fault type.

[0029] Therefore, the embodiment of the present application can discover the fault of the all-flash storage system in time and accurately match the diagnosis mechanism, which helps to quickly start the fault processing, reduces the influence of the system fault, and guarantees the stable operation of the storage system.

[0030] Optionally, in an embodiment of the present application, before detecting whether a fault occurs in the all-flash storage system during operation, the method further includes: classifying a plurality of thin pool faults of the all-flash storage system, and matching the fault self-repair mechanism corresponding to different types of thin pool faults; and importing the fault self-repair mechanism into the fault automatic repair model corresponding to the all-flash storage system, wherein the fault self-repair mechanism includes a fault detection mechanism, a fault diagnosis mechanism, a fault repair mechanism, and a fault feedback mechanism.

[0031] It should be noted that the embodiment of the present application can first establish an automatic repair mechanism and framework to classify and process the faults of the all-flash storage system; secondly, the embodiment of the present application can perform corresponding warning uploading, identification diagnosis, and repair processing operations for each type of fault, and register or import the detection, diagnosis, repair, feedback, and other operations of each fault into the mechanism and framework of the automatic repair system (i.e., the pre-constructed fault automatic repair model corresponding to the all-flash storage system).

[0032] The following describes the processing process of the thin pool fault full cycle with reference to the accompanying drawings.

[0033] Figure 2 The execution logic diagram for the full-cycle processing of the thin pool fault in the embodiment of the present application is shown in FIG. 1. Figure 2 As shown in FIG. 1, the process of the full-cycle processing of the thin pool fault in the embodiment of the present application is as follows:

[0034] S201: thin pool fault classification;

[0035] S202: thin pool fault detection;

[0036] S203: thin pool fault diagnosis;

[0037] S204: repair of different thin pool faults.

[0038] Therefore, the embodiments of the present application can well obtain and call the processing and repair mechanism of the corresponding fault by classifying the faults of the all-flash storage system and registering or importing the detection, diagnosis, repair, feedback and other operations of each type of fault into the automatic fault repair model.

[0039] Optionally, in an embodiment of the present application, a plurality of thin pool faults of the all-flash storage system are classified, including: collecting historical fault data of the corresponding thin pool of the all-flash storage system, wherein the historical fault data of the thin pool includes running parameter change data, fault phenomenon description information and fault category label; performing standardization processing on the historical fault data of the thin pool to obtain corresponding standard historical fault data, and constructing a corresponding fault feature data set based on the standard historical fault data; dividing the fault feature data set into a training subset and a validation subset, and using the training subset to train a pre-constructed fault classification initial model, and inputting the validation subset into the trained fault classification initial model to adjust the network parameters of the fault classification initial model through the validation subset to generate a target fault classification model; obtaining the to-be-classified fault data of a plurality of thin pool faults corresponding to the all-flash storage system, and extracting the feature vector corresponding to the to-be-classified fault data, and inputting the feature vector into the target fault classification model to output the fault types corresponding to the plurality of thin pool faults.

[0040] Specifically, the embodiments of the present application can first collect the historical fault data of the thin pool of the all-flash storage system, covering running parameter change data (such as storage capacity usage rate fluctuation, IOPS (Input / Output Operations Per Second) peak value change), fault phenomenon description information (such as read / write response delay, data write failure prompt) and fault category label (such as space allocation failure, metadata damage).

[0041] Secondly, the embodiments of the present application can perform standardization processing on the historical fault data. As a kind of can be realized mode, the embodiments of the present application can uniformly convert different format parameter data into numerical type (such as converting "high / medium / low" delay into quantization value in 0-1 interval), perform word segmentation and vector encoding on text description information, remove abnormal values and repeated data to form standard historical fault data; then, the embodiments of the present application can extract corresponding features (such as parameter change rate, fault duration) from the standard historical fault data to construct a fault feature data set.

[0042] Thirdly, the embodiments of the present application can divide the data set into a training subset and a validation subset according to a 7:3 ratio, and select an improved random forest as a fault classification initial model to improve the classification accuracy by increasing the number of decision trees.

[0043] After that, the embodiment of the present application can train the model with the training subset, input the validation subset into the model after each iteration, calculate the classification accuracy, and adjust the network parameters such as tree node split threshold in reverse, until the accuracy of the model on the validation set is stable above 90%, and the target fault classification model is generated.

[0044] After obtaining the fault data to be classified, the embodiment of the present application can extract the feature vector (such as the capacity usage rate of a certain fault rising by 80%, IOPS falling by 50%) of the fault data, input the feature vector into the target fault classification model, and the model outputs the corresponding fault type (such as determining as "space allocation failure") by comparing the feature vector with the feature patterns of each type of fault obtained by training.

[0045] Therefore, the embodiment of the present application realizes the accurate classification of the simplified pool fault through standardization processing and model training, improves the fault identification efficiency and accuracy, and provides guarantee for the stable operation of the all-flash storage system.

[0046] Optionally, in an embodiment of the present application, before the fault self-repair mechanism is introduced into the pre-constructed fault automatic repair model corresponding to the all-flash storage system, it further includes: receiving the fault information corresponding to the all-flash storage system pushed by the business layer in the all-flash storage system through the preset fault receiving unit, and performing fault analysis according to the fault information by using the preset fault diagnosis unit to obtain corresponding fault analysis data; sending the fault analysis data to the preset fault feedback unit, so as to perform fault repair processing on the all-flash storage system according to the fault analysis data by using the fault feedback unit, so as to generate corresponding fault repair information; sending the fault repair information to the preset fault feedback unit, and sending the fault analysis data to the business layer by using the fault feedback unit, so as to construct the fault automatic repair model.

[0047] It should be noted that the automatic repair framework (i.e. the fault automatic repair model) in the embodiment of the present application is based on the storage system, and mainly includes a framework state machine, a fault receiving unit, a fault diagnosis unit, a fault repair unit and a fault feedback unit.

[0048] Among them, the framework state machine is mainly responsible for the interaction with the business unit of the all-flash storage system (i.e. the business layer in the all-flash storage system) and the driving and transfer of fault processing; the fault receiving unit mainly receives the fault information pushed by the business unit; the fault diagnosis unit is mainly responsible for analyzing and processing the fault; the fault repair unit is mainly responsible for repairing the fault; the fault feedback unit is mainly responsible for feeding back the corresponding fault repair information to the all-flash business unit; the all-flash business unit performs the next round of fault processing according to the feedback fault repair information.

[0049] In addition, the fault automatic repair model in the embodiment of the present application provides a registration interface of a fault receiving unit, a fault diagnosis unit, a fault repair unit and a fault feedback unit, so that the all-flash service unit performs corresponding registration operation through the above-mentioned corresponding registration interface.

[0050] Therefore, the embodiment of the present application constructs the fault automatic repair model, so that the functional units cooperatively process the fault of the all-flash storage system, and provides a corresponding registration interface to facilitate the service unit to access, thereby effectively improving the fault processing efficiency and ensuring the stability of the system.

[0051] Optionally, in an embodiment of the present application, the fault self-repair mechanism is introduced into the fault automatic repair model of the all-flash storage system, including: performing a thin pool fault detection operation on the all-flash storage system to obtain a corresponding detection result, and determining a fault detection mechanism according to the thin pool fault detection process, and introducing the fault detection mechanism into the fault automatic repair model, wherein the thin pool fault includes a log volume fault, a configuration management fault, a metadata fault, a deduplication fault, a compression fault and a garbage collection fault; judging whether the all-flash storage system has a thin pool fault according to the detection result, so that in the case that the all-flash storage system has a thin pool fault, a thin pool fault diagnosis operation is performed to generate corresponding diagnosis data, and a fault diagnosis mechanism is determined based on the thin pool fault diagnosis process, and the fault diagnosis mechanism is introduced into the fault automatic repair model; based on the diagnosis data, performing a fault repair action corresponding to the thin pool fault, and generating fault feedback information after fault repair, so as to perform fault detection operation on the all-flash storage system again by using the fault feedback information, and determining a fault repair mechanism and a fault feedback mechanism according to the fault repair process and the fault feedback information generation process, and introducing the fault repair mechanism and the fault feedback mechanism into the fault automatic repair model.

[0052] In the embodiment of the present application, the all-flash storage system mainly includes log volume, thin pool state management, metadata management, garbage collection, deduplication module, compression module and other sub-components. In actual execution process, the sub-components of the all-flash storage system of the embodiment of the present application may have log volume fault, configuration management fault, metadata fault, deduplication fault, compression fault and garbage collection fault.

[0053] Specifically, the process of processing and registering different faults by the embodiment of the present application is as follows:

[0054] 1. Log volume fault:

[0055] LSA (Log-Structured Writing) is a storage management technology based on log-structured writing, widely used in all-flash storage systems (such as Pure Storage, Dell PowerStore) and distributed storage (such as Ceph Bluestore), aiming to optimize the write performance of flash and prolong the life of SSD (Solid State Drive).

[0056] The core idea of LSA is to sequentially append all write operations to the log, rather than directly overwrite the original data, thereby avoiding performance degradation and flash wear problems caused by random writes, etc. The core functions include space management (allocation, balancing, etc.), IO (Input Output) write disk control, and IO traffic control. Common failure problems include space allocation depletion causing IO flushing, and latency jitter.

[0057] The process of registering the fault detection, fault diagnosis, and fault handling mechanisms for log volume failure is as follows:

[0058] (1) Fault detection registration: if the latency of LSA flushing to the disk exceeds 5 times the average value, initiate fault detection;

[0059] (2) Fault diagnosis registration: diagnose the process of LSA flushing to the disk latency to generate corresponding diagnosis data;

[0060] (3) Fault repair: based on the diagnosis data, execute the repair action corresponding to the log volume failure, and generate the fault feedback information after fault repair, LSA does not wait for aggregation and flushes immediately.

[0061] 2. Configuration management failure:

[0062] Configuration management mainly involves volume management and running state management of thin pools. Configuration management failure mainly refers to the abnormality of thin pool state machine, such as internal state field check abnormality and state stuck, etc. Each type of configuration management failure and related diagnosis and repair mechanisms are registered.

[0063] 3. Metadata failure:

[0064] Metadata management is the core of the thin pool storage system, mainly responsible for the mapping of logical address to physical address, and its stability and efficiency determine the pros and cons of the thin pool storage system. Due to the involvement of the deduplication feature (the need to manage the organization relationship of a physical address corresponding to multiple logical addresses), space recycling, system power failure data repair and other core functions of spatial distribution design, the spatial organization complexity is extremely high, and the failure of metadata is more likely to cause fatal problems. In the process of handling such failures, reverse thinking can be used to analyze the problem results uniformly to repair the repairable failures, thereby greatly improving the reliability of the storage.

[0065] (1) Fault registration: an exception occurs in the metadata process (such as accessing an empty pointer), and detection is initiated;

[0066] (2) Fault detection registration: an exception occurs in the metadata read cache process (such as accessing an empty pointer), and detection is initiated. The empty pointer exception is a default fault, and directly enters the diagnosis process;

[0067] (3) Fault diagnosis registration: diagnose the access to the empty pointer that occurs in the read cache process. The diagnosis logic is whether it can be avoided from a business perspective. Since the process accesses the read cache, it can be avoided from a business perspective, such as releasing all caches and directly reading data from the disk.

[0068] (4) Fault repair: For the above fault diagnosis, perform the corresponding repair action, release all caches, directly read data from the disk, and at the same time, report an alarm to reduce the risk of the system;

[0069] 4. Deduplication fault:

[0070] Deduplication is an important feature of data reduction, which leads to high complexity of data management. This module mainly handles the faults caused by the deduplication feature. The deduplication module is prone to problems due to its high complexity, but it has very reliable repair means and can be avoided. For example, if an unrepairable problem is encountered, the deduplication function can be turned off, and the normal thin pool logic (without data deduplication processing) can be run to avoid business downtime.

[0071] 5. Compression fault:

[0072] Compression is an important feature of data reduction, and this module mainly handles the faults caused by the compression feature. The function processing of the compression module is similar to that of deduplication. Once an unrepairable problem is encountered, the compression function can be turned off, and the normal thin pool logic (without data compression) can be run to avoid business downtime.

[0073] 6. Garbage collection fault: faults that occur during space recycling.

[0074] As an implementable way, the embodiment of the application takes the metadata failure of the thin pool as an example to analyze the failure registration process in detail, as follows.

[0075] 1. The thin pool service unit registers the alarm, and when the metadata failure occurs, the alarm can be captured by the failure automatic detection and diagnosis mechanism.

[0076] 2. The failure diagnosis mechanism is also registered. It should be noted that the failure diagnosis mechanism (i.e. the failure diagnosis positioning process after the failure occurs) is provided by the development and maintenance personnel of the service unit; in addition, the failure repair mechanism, failure result feedback mechanism, etc. also need to be registered in the embodiment of the application.

[0077] 3. During the operation of the all-flash storage system, once the failure alarm is captured, the embodiment of the application can call the diagnosis interface of the service unit for diagnosis. After diagnosis, if the cause is found to be that the metadata between the controllers do not match, the corresponding analysis result is generated, and the state machine is controlled to enter the next repair action according to the analysis result. If there is a corresponding repair action, the corresponding repair action is executed, otherwise it is fed back that the repair action cannot be executed, and the alarm is retained.

[0078] For example, in a dual-control storage system, one controller handles an exception caused by inserting metadata due to deduplication, and under normal circumstances, the controller cannot handle the exception and hands over to the other controller for processing. When the other controller also has a problem, it is likely to cause exceptions of both controllers and business downtime. However, if similar failures are classified and processed, if a failure is detected, detection is performed, and then normal metadata is inserted to avoid the problem through the failure repair mechanism (such as stopping the deduplication operation), and an alarm is reported for processing, thereby effectively avoiding the risk of business downtime.

[0079] It should be noted that the above method of avoiding business downtime needs to understand and summarize the business in depth, and the repair is performed through the import test scenario, diagnosis method and repair method, which starts from the summarized failure scenario and automatic repair, but does not guarantee that the system will not have problems. In addition, in the registration process, the embodiment of the application does not traverse and register all failures, but registers typical case scenarios for each functional unit.

[0080] Therefore, the embodiment of the application classifies the thin pool failure types, and registers each failure according to the failure detection, failure diagnosis, failure repair and failure processing result feedback process mechanism, so that the all-flash storage system has stronger storage fault tolerance performance and anti-interference ability.

[0081] Optionally, in an embodiment of the present application, the thin pool fault detection operation is performed on the all-flash storage system, comprising: collecting current running parameters and current load parameters of the all-flash storage system, and generating corresponding load state data according to the current load parameters; calling a preset load classification rule through the load state data, and classifying the load state of the all-flash storage system based on the load classification rule to obtain a corresponding load state classification result; determining the current load level of the all-flash storage system according to the load state classification result, setting a fault detection threshold corresponding to the current load level, and judging whether the current running parameters are greater than the fault detection threshold; if the current running parameters are greater than the fault detection threshold, it is determined that the all-flash storage system has a thin pool fault; if the current running parameters are less than or equal to the fault detection threshold, it is determined that the all-flash storage system does not have a thin pool fault.

[0082] As an implementable way, the specific process of the fault detection performed by the embodiment of the present application is as follows:

[0083] 1. Real-time collection of current running parameters and current load parameters of the all-flash storage system, wherein the current running parameters cover storage unit response delay, data read / write error frequency, etc., and the current load parameters include active task quantity, data transmission bandwidth occupancy rate, etc., and the two types of parameters are integrated to generate corresponding load state data;

[0084] 2. Calling a preset load classification rule through the load state data, which is formulated in advance in combination with load characteristics of the system in different application scenarios and contains multi-dimensional judgment indexes, based on which the load state of the all-flash storage system is classified to obtain a corresponding load state classification result;

[0085] 3. Determining the current load level of the all-flash storage system according to the load state classification result, calling a fault detection threshold benchmark matched with the load level, fine-tuning the threshold combined with recent fault history data of the system, setting a fault detection threshold corresponding to the current load level, and then comparing the current running parameters with the fault detection threshold one by one to judge whether the current running parameters are greater than the fault detection threshold;

[0086] 4. If the current running parameters are greater than the fault detection threshold, further associating the parameter abnormal type with the corresponding relationship of the thin pool fault, accurately determining the type of the thin pool fault existing in the all-flash storage system; if the current running parameters are less than or equal to the fault detection threshold, recording the current running state data as a reference basis for subsequent optimization of the load classification rule, and determining that the all-flash storage system does not have a thin pool fault.

[0087] Therefore, the embodiments of the present application can more accurately detect faults, can adapt to different load states, can locate fault types, can accumulate data to optimize rules, can effectively reduce fault false alarm and missed alarm rates, and can improve the reliability and practicality of fault detection of the all-flash storage system.

[0088] Optionally, in an embodiment of the present application, whether the all-flash storage system has a thin pool fault is judged according to the detection result, so that, in the case that the all-flash storage system has a thin pool fault, a thin pool fault diagnosis operation is performed to generate corresponding diagnosis data, and a fault diagnosis mechanism is determined based on the thin pool fault diagnosis process, and the fault diagnosis mechanism is introduced into the fault automatic repair model, including: when performing a log volume fault diagnosis on the all-flash storage system, judging the node running mode of the all-flash storage system; if the all-flash storage system is in a dual-node running mode, it is determined that the all-flash storage system does not have a log volume fault; if the all-flash storage system is in a single-node running mode, detecting the corresponding back-end latency of the all-flash storage system, and judging whether the back-end latency is less than a preset latency threshold; when the back-end latency is less than the latency threshold, stopping the log volume fault diagnosis operation; when the back-end latency is greater than or equal to the latency threshold, performing a logical diagnosis operation on the all-flash storage system to obtain corresponding logical diagnosis data, and judging whether the logical diagnosis data meets a preset aggregation requirement; if the logical diagnosis data does not meet the aggregation requirement, stopping the log volume fault diagnosis operation; if the logical diagnosis data meets the aggregation requirement, performing a corresponding fault repair operation on the log volume fault, and determining a fault diagnosis mechanism corresponding to the log volume fault based on the log volume fault diagnosis process of the all-flash storage system, and introducing the fault diagnosis mechanism into the fault automatic repair model.

[0089] It should be noted that when performing a log volume fault diagnosis on the all-flash storage system, the embodiments of the present application first need to accurately judge the current node running mode of the system, for example, the embodiments of the present application can read parameters such as “node number” and “cluster communication state” in the system configuration file, and if the parameters show that only a single node is in an active state and there is no communication record of a standby node, it is determined that the single-node running mode. Then, through the built-in latency monitoring tool of the system, the data transmission latency between the back-end storage device (such as an SSD hard disk) and the controller is collected, which is sampled once every 100 ms, and the average value of 10 consecutive samplings is taken as the final back-end latency.

[0090] If it is detected that both nodes are in an active state and there is data synchronization communication, it is determined that the dual-node running mode. At this time, the log volume does not need to be cooperated by multiple nodes, and the “system normal” result is returned and the detection is stopped, and it is directly determined that there is no log volume fault.

[0091] Secondly, the embodiment of the present application compares the time delay with a preset time delay threshold (such as 50 ms, which is set according to the read-write performance requirement of the all-flash storage system): if the time delay is greater than or equal to 50 ms, it indicates that there is a transmission bottleneck in the back-end hardware, and continuing the diagnosis may lead to misdiagnosis, so the log volume fault diagnosis operation is stopped.

[0092] If the back-end time delay is less than 50 ms, it enters the logical diagnosis stage. Specifically, the embodiment of the present application can call the system log analysis module to extract the logical diagnosis data such as "data aggregation frequency" and "flushing progress" of the log volume in the past one hour, and judge whether the data is not technically flushed due to long-time non-aggregation (such as more than 30 minutes of non-aggregation operation). That is, it checks whether the logical diagnosis data meets the aggregation requirement of "at least once data aggregation every 15 minutes and flushing progress of 100%".

[0093] If not, it indicates that there is an abnormal data processing in the logical layer, and the diagnosis is stopped; if yes, it starts the fault repair operation, such as triggering data forced aggregation and flushing, repairing log volume metadata errors, etc. At the same time, the diagnosis process is sorted out, and the log volume fault diagnosis mechanism of "hardware transmission detection first, and then logical data verification" is determined. The judgment logic, threshold parameters, etc. of the mechanism are packaged as a standardized module, and are registered or imported into the fault automatic repair model for subsequent diagnosis.

[0094] The following describes the execution logic of the log volume fault diagnosis of the all-flash storage system through the accompanying drawings.

[0095] Figure 3 The execution logic of the log volume fault diagnosis of the all-flash storage system is shown in the accompanying drawings. As shown in Figure 3 The execution process of the log volume fault diagnosis of the all-flash storage system by the embodiment of the present application is described as follows:

[0096] S301: Simplified pool log volume flushing timeout detection;

[0097] S302: Start fault diagnosis operation;

[0098] S303: Judge whether the all-flash storage system is single-node running. If the all-flash storage system is single-node running, go to S304, otherwise go to S307;

[0099] S304: Judge whether the back-end time delay of the all-flash storage system is less than the time delay threshold. If the back-end time delay is less than the time delay threshold, go to S305, otherwise go to S307;

[0100] S305: Judge whether the all-flash storage system meets the preset aggregation requirement. If the aggregation requirement is met, go to S306, otherwise go to S307;

[0101] S306: Start fault repair operation;

[0102] S307: The fault diagnosis ends, and it is determined that there is no log volume fault in the full flash storage system.

[0103] Therefore, the embodiment of the present application can effectively avoid invalid diagnosis by hierarchical diagnosis according to the node mode, thereby ensuring accurate and efficient log volume fault diagnosis and helping the stable operation of the full flash storage system.

[0104] Optionally, in an embodiment of the present application, whether the full flash storage system has a fault during operation is detected, and in the case where it is detected that the full flash storage system has a fault, the fault type of the fault is determined to call the fault diagnosis mechanism corresponding to the fault type from the pre-constructed fault automatic repair model according to the fault type, comprising: based on a pre-set timer polling strategy, a pre-set system state machine is controlled to detect whether the full flash storage system has a fault during operation; when the full flash storage system has a fault during operation, the fault diagnosis mechanism corresponding to the fault in the fault automatic repair model is called, and the fault diagnosis operation is performed on the fault through the fault diagnosis mechanism to diagnose the fault cause of the full flash storage system; when the full flash storage system has no fault during operation, the system state machine is used again to detect whether the full flash storage system has a fault during operation.

[0105] In actual execution, the embodiment of the present application can use a pre-set timer polling strategy to configure the system state machine to perform operation state detection on the full flash storage system at fixed time intervals (such as once every 2 seconds).

[0106] Specifically, the embodiment of the present application can collect core indicators such as CPU (Central Processing Unit, central processor) usage rate, memory occupation, IO response time, compare with a pre-set normal threshold range, and listen to error codes thrown by the system kernel (such as IO timeout error 0x0001, disk read / write failure 0x0002) to comprehensively judge whether a fault occurs.

[0107] In the embodiment of the present application, all possible fault types (such as log volume fault, configuration management fault, metadata fault, deduplication fault, compression fault, and garbage collection fault) are pre-imported into the automatic detection system, each fault type is associated with a unique identifier and a corresponding fault diagnosis mechanism. When the system state machine detects a fault, the automatic detection system immediately calls the matching fault diagnosis mechanism in the registration library according to the fault identifier. During the diagnosis process, the embodiment of the present application can record fault feature data in real time to generate a diagnosis result containing fault level (such as warning / severe), impact range (such as single node / whole system), and possible cause. If no fault is detected, the system state machine will re-execute the detection process according to the original polling period to ensure continuous monitoring.

[0108] Therefore, the embodiments of the present application realize fast fault detection and accurate diagnosis by combining timing polling with pre-registration mechanism, thereby guaranteeing the timeliness of fault response of the all-flash storage system and improving the system reliability and stability.

[0109] In step S102, the fault diagnosis mechanism is used to diagnose the fault cause of the all-flash storage system, and whether the all-flash storage system meets the preset repair condition is determined according to the fault cause.

[0110] Further, the embodiments of the present application can monitor the system running state in real time during the running of the all-flash storage system to identify whether a fault occurs; if a fault is monitored to occur, the corresponding fault diagnosis mechanism pre-configured by the system is immediately invoked, and targeted diagnosis is carried out on the fault according to the diagnosis logic and operation steps in the mechanism to obtain the fault cause corresponding to the fault, and whether the all-flash storage system meets the preset repair condition is determined according to the fault cause.

[0111] Therefore, the embodiments of the present application can quickly locate the key information of the fault by monitoring the fault in real time and invoking the corresponding diagnosis mechanism for accurate diagnosis, thereby gaining time for subsequent fault repair and effectively guaranteeing the stable running of the all-flash storage system.

[0112] In step S103, if the preset repair condition is met, the fault cause is input into the fault automatic repair model to invoke the fault repair mechanism corresponding to the fault cause, and the all-flash storage system is repaired by the fault repair mechanism, otherwise, manual repair prompting is performed.

[0113] Then, the embodiments of the present application can check whether the fault meets the automatic repair condition according to the fault cause: if yes, the fault cause is input into the fault automatic repair model to invoke the corresponding repair mechanism from the fault automatic repair model to automatically execute the repair operation; if not, the manual intervention process is triggered to handle the fault, thereby enhancing the fault repair capability and stability of the all-flash storage system.

[0114] It can be understood that the embodiments of the present application establish the system for automatic repair of the thin pool fault by combining the business characteristics of the thin pool, the fault repair method and the automatic repair mechanism framework, classify the fault types of the thin pool, and register each fault according to the processing mechanisms such as fault detection, fault diagnosis, fault repair and fault processing result feedback. After the system runs, the fault is automatically handled according to the injected fault diagnosis and repair mechanism, so that the fault can be repaired in the fastest and most effective way, and the continuity of the business is maximally guaranteed.

[0115] Optionally, in an embodiment of the present application, if the preset repair condition is met, the fault reason is input into the fault automatic repair model to call the fault repair mechanism corresponding to the fault reason, and the all-flash storage system is repaired by the fault repair mechanism, otherwise, the manual repair prompt is performed, including: based on the fault reason, it is judged whether the fault meets the preset repair condition; if the fault meets the preset repair condition, the corresponding fault repair mechanism in the fault automatic repair model is called to execute the corresponding fault repair operation on the all-flash storage system by the fault repair mechanism to generate the corresponding actual fault repair information; if the fault does not meet the preset repair condition, the corresponding service processing interruption operation is performed on the all-flash storage system, and the fault report corresponding to the all-flash storage system is generated, and the fault is sent to the target user end, so that the target user performs manual repair operation on the fault according to the fault report.

[0116] In the specific implementation process, first, according to the fault diagnosis result (including fault type, influence range, severity level), it is judged whether the preset repair condition is met, for example, the fault such as "single node log volume data synchronization exception" only affects the local and has a mature automatic repair scheme, and it is determined that the requirement is met; the fault such as "multi-node communication complete interruption" has no automatic repair path, and it is determined that the requirement is not met.

[0117] If the repair condition is met, the matched fault repair mechanism (such as log volume data forced synchronization mechanism) can be called from the fault automatic repair model, and the operation (such as triggering inter-node data incremental synchronization) is executed through the repair action interface built in the mechanism.

[0118] After the repair is completed, the system state machine monitors the core indicators such as IO response time in real time, checks whether the system is repaired normally, if the indicators return to the normal threshold, immediately notifies the registered core business unit in the all-flash storage system to repair the business processing, if the indicators are still abnormal, notifies the core module that the fault is not eliminated, continues to send the alarm information (such as pushing to the operation and maintenance management platform) or suspends the associated business to prevent the fault from spreading.

[0119] If the fault does not meet the repair condition (special fault scenario), the automatic repair operation is not performed, the business processing interruption process (such as suspending the read-write service of the fault node) is directly triggered, and the fault report containing the fault details and the diagnosis process is generated and pushed to the target user end (such as the management terminal of the operation and maintenance personnel), and waits for manual intervention repair.

[0120] It should be noted that in the embodiment of the present application, whether the fault repair is successful or not, after the current fault is processed, the system state machine immediately switches to the polling mode, and detects the next potential fault according to the preset period.

[0121] Therefore, the embodiments of the present application can automatically identify, detect, repair and feedback after a fault occurs, so as to complete automatic fault repair, thereby greatly shortening the fault repair time, avoiding business interruption or interruption time exceeding the business requirements caused by the fault, and enabling the business to continue to be executed without being perceived by the customer.

[0122] Optionally, in an embodiment of the present application, the target user performs manual repair operation on the fault according to the fault report, including: analyzing the fault report to obtain corresponding report analysis data, and extracting the fault type, fault influence range, diagnosis result and service interruption state information in the report analysis data; determining manual repair target data based on the fault type, fault influence range, diagnosis result and service interruption state information; determining a corresponding manual repair scheme according to the manual repair target data, so as to perform manual repair operation on the fault based on the manual repair scheme.

[0123] It should be noted that after the target user (such as an operation and maintenance personnel) receives the fault report, the embodiments of the present application can first analyze it by the built-in report analysis tool to convert the unstructured fault description text into standardized report analysis data, and accurately extract key information such as fault type (specifically, log data damage), influence range (involving data records of the thin pool state management module), diagnosis result (showing checksum mismatch), and service interruption state (part of read-write function is limited).

[0124] Based on the above key information, the embodiments of the present application can determine the manual repair target data, for example, for the log volume fault, the target is to repair the damaged log, restore the normal data interaction with the thin pool state management module, and restore the limited read-write service. Then, the embodiments of the present application can formulate a corresponding manual repair scheme according to the target data, if it is log data damage, the scheme may include repairing data from backup logs, checking the interaction link with the thin pool state management module, and gradually restarting related services, and performing manual repair operation according to the scheme.

[0125] Therefore, the embodiments of the present application can accurately formulate a repair scheme by analyzing the report in combination with the fault characteristics of each subcomponent, thereby improving the manual repair efficiency and reducing the impact of the fault on the all-flash storage system.

[0126] Optionally, in an embodiment of the present application, further comprising: determining the occurrence frequency and business impact degree of different types of thin pool faults in the fault automatic repair model, and marking the priority of different types of thin pool faults according to the occurrence frequency and business impact degree to generate a corresponding fault priority list; adjusting the timer polling strategy through the fault priority list, and processing new faults generated by the all-flash storage system during operation based on the fault automatic repair model and the adjusted timer polling strategy, and recording the detection time consumption, diagnosis time consumption and repair time consumption corresponding to the new faults; adjusting the priority of different types of thin pool faults and the timer polling strategy based on the detection time consumption, diagnosis time consumption and repair time consumption, so as to re-perform corresponding fault handling operations on the all-flash storage system.

[0127] In actual execution, since the all-flash storage system includes sub-components such as log volumes, various faults such as log volume faults and metadata faults may occur, and therefore the embodiment of the present application can count the occurrence frequency (such as the number of occurrences per month) and business impact degree (such as the range of business affected and the duration of interruption) of each component fault (such as write failure of log volume).

[0128] Secondly, the embodiment of the present application can mark the priority of different types of thin pool faults, such as marking the metadata fault with high frequency and causing core business interruption as P1, and marking the compression fault with low frequency and slight impact as P3, to form a priority list, and accordingly adjust the timer polling strategy, with the detection interval of P1 fault shortened to 1 second and the detection interval of P3 fault extended to 10 seconds.

[0129] When processing new faults, the embodiment of the present application can record the detection, diagnosis and repair time consumption, and periodically analyze the recorded data, and if the time consumption of a certain P2 fault becomes longer, the priority is raised and the detection interval is shortened, otherwise the priority is lowered, to dynamically optimize fault handling.

[0130] Thus, the embodiment of the present application dynamically adjusts the priority and polling strategy, thereby focusing on key faults and improving the efficiency and pertinence of fault handling of the all-flash storage system, and ensuring the stability of the system.

[0131] In addition, the embodiment of the present application can also analyze the fault correlation, and dynamically adjust the priority in combination with the correlation between log volume and metadata faults, and introduce predictive maintenance to predict the fault trend based on historical data, and optimize the polling strategy in advance to improve the processing foresight. The specific process is as follows:

[0132] Step 1: Construct a fault correlation model, collect historical fault data such as log volume faults and metadata faults in the all-flash storage system, calculate the correlation between different faults (e.g., through co-occurrence frequency and conditional probability), classify them into strong correlation (correlation degree ≥ 0.8), medium correlation (0.4 < correlation degree < 0.8), and weak correlation (correlation degree ≤ 0.4) levels, and output a fault correlation level table to provide a correlation basis for dynamic priority adjustment;

[0133] Step 2: Based on the fault association level table, adjust the fault priority in the simplified pool; in the strongly associated fault group, if the core fault is marked as P1, the associated faults are simultaneously promoted by 1 level; the weakly associated faults retain their original priority, and output the adjusted fault priority list to provide a priority benchmark for polling strategy optimization;

[0134] Step 3: Establish a fault prediction model. Input historical fault data and system operating parameters (such as load and temperature). Use an LSTM network to predict the probability and trend of different types of faults in the next 7 days and output the fault prediction trend results. Combine the results with the priority list in Step 2 to optimize the timer polling strategy (such as shortening the high-frequency fault detection interval by 20%) and output the optimized polling strategy.

[0135] Therefore, the embodiments of this application construct a fault association model to classify association levels, dynamically adjust fault priorities, and combine LSTM to predict fault trends to optimize the polling strategy, thereby accurately focusing on key faults and improving the foresight and efficiency of fault handling in all-flash storage systems.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0137] Embodiments of this application also provide a fault repair device for an all-flash storage system.

[0138] like Figure 4 As shown, the fault repair device 10 for the all-flash storage system includes: a fault detection module 100, a fault diagnosis module 200, and a fault repair module 300.

[0139] The fault detection module 100 is used to detect whether a fault occurs in the all-flash storage system during operation, and when a fault is detected in the all-flash storage system, to determine the fault type, and to call the corresponding fault diagnosis mechanism from the pre-built automatic fault repair model according to the fault type.

[0140] The fault diagnosis module 200 is configured to diagnose a fault cause of the all-flash storage system by using a fault diagnosis mechanism, and determine whether the all-flash storage system meets a preset repair condition according to the fault cause.

[0141] The fault repair module 300 is configured to input the fault cause into a fault automatic repair model to call a fault repair mechanism corresponding to the fault cause, and perform a repair operation on the all-flash storage system by using the fault repair mechanism if the all-flash storage system meets the preset repair condition, or perform a manual repair prompt if the all-flash storage system does not meet the preset repair condition.

[0142] Optionally, in an embodiment of the present application, the fault repair device 10 of the all-flash storage system further includes a classification module and an import module.

[0143] The classification module is configured to classify a plurality of thin pool faults of the all-flash storage system before detecting whether the all-flash storage system has a fault during operation, and match a fault self-repair mechanism corresponding to different types of thin pool faults.

[0144] The import module is configured to import the fault self-repair mechanism into a fault automatic repair model corresponding to the all-flash storage system, where the fault self-repair mechanism includes a fault detection mechanism, a fault diagnosis mechanism, a fault repair mechanism, and a fault feedback mechanism.

[0145] Optionally, in an embodiment of the present application, the import module includes a detection mechanism import unit, a diagnosis mechanism import unit, and a feedback mechanism import unit.

[0146] The detection mechanism import unit is configured to perform a thin pool fault detection operation on the all-flash storage system to obtain a corresponding detection result, determine a fault detection mechanism according to a thin pool fault detection process, and import the fault detection mechanism into the fault automatic repair model, where the thin pool fault includes a log volume fault, a configuration management fault, a metadata fault, a deduplication fault, a compression fault, and a garbage collection fault.

[0147] The diagnosis mechanism import unit is configured to determine whether the all-flash storage system has a thin pool fault according to the detection result, perform a thin pool fault diagnosis operation to generate corresponding diagnosis data in a case where the all-flash storage system has a thin pool fault, determine a fault diagnosis mechanism based on a thin pool fault diagnosis process, and import the fault diagnosis mechanism into the fault automatic repair model.

[0148] The feedback mechanism introduction unit is configured to perform a fault repair action corresponding to the simplified pool fault based on the diagnostic data, and generate fault feedback information corresponding to the fault after repair, so as to perform a fault detection operation on the all-flash storage system again by using the fault feedback information, and determine a fault repair mechanism and a fault feedback mechanism according to the fault repair process and the fault feedback information generation process, and introduce the fault repair mechanism and the fault feedback mechanism into the automatic fault repair model.

[0149] Optionally, in an embodiment of the present application, the fault detection module comprises a control unit, a diagnosis unit and a re-detection unit.

[0150] The control unit is configured to control the preset system state machine to detect whether a fault occurs in the all-flash storage system during operation based on a preset timer polling strategy.

[0151] The diagnosis unit is configured to call a fault diagnosis mechanism corresponding to the fault in the automatic fault repair model when a fault occurs in the all-flash storage system during operation, and perform a fault diagnosis operation on the fault by using the fault diagnosis mechanism, so as to diagnose the fault cause of the all-flash storage system.

[0152] The re-detection unit is configured to re-detect whether a fault occurs in the all-flash storage system during operation by using the system state machine when no fault occurs in the all-flash storage system during operation.

[0153] Optionally, in an embodiment of the present application, the fault repair module comprises an analysis unit, a calling unit and a sending unit.

[0154] The analysis unit is configured to determine whether the fault satisfies a preset repair condition based on the fault cause.

[0155] The calling unit is configured to call a fault repair mechanism corresponding to the fault in the automatic fault repair model if the fault satisfies the preset repair condition, so as to perform a corresponding fault repair operation on the all-flash storage system by using the fault repair mechanism, and generate actual fault repair information corresponding to the fault.

[0156] The sending unit is configured to perform a corresponding service processing interruption operation on the all-flash storage system and generate a fault report corresponding to the all-flash storage system if the fault does not satisfy the preset repair condition, and send the fault to a target user end, so that the target user performs a manual repair operation on the fault according to the fault report.

[0157] Optionally, in an embodiment of the present application, the fault repair device 10 of the all-flash storage system further comprises a priority marking module, a polling module and an adjustment module.

[0158] The priority marking module is configured to determine the occurrence frequency and business impact degree of different types of thin pool faults in the fault automatic repair model, and mark priorities of the different types of thin pool faults according to the occurrence frequency and the business impact degree, so as to generate a corresponding fault priority list.

[0159] The polling module is configured to adjust a timer polling strategy according to the fault priority list, and handle a new fault generated by the all-flash storage system during operation based on the fault automatic repair model and the adjusted timer polling strategy, and record detection time consumption, diagnosis time consumption and repair time consumption corresponding to the new fault.

[0160] The adjusting module is configured to adjust the priorities of the different types of thin pool faults and the timer polling strategy based on the detection time consumption, the diagnosis time consumption and the repair time consumption, so as to perform corresponding fault handling operations on the all-flash storage system again.

[0161] Optionally, in an embodiment of the present application, the diagnosis mechanism importing unit comprises a first judgment subunit, a first judgment subunit, a second judgment subunit, a first stop diagnosis subunit, a third judgment subunit, a second stop diagnosis subunit and an aggregation subunit.

[0162] The first judgment subunit is configured to judge a node running mode of the all-flash storage system when diagnosing a log volume fault of the all-flash storage system.

[0163] The first judgment subunit is configured to judge that the all-flash storage system does not have a log volume fault if the all-flash storage system is in a dual-node running mode.

[0164] The second judgment subunit is configured to detect a back-end latency corresponding to the all-flash storage system if the all-flash storage system is in a single-node running mode, and judge whether the back-end latency is less than a preset latency threshold.

[0165] The first stop diagnosis subunit is configured to stop a log volume fault diagnosis operation when the back-end latency is less than the latency threshold.

[0166] The third judgment subunit is configured to perform a logical diagnosis operation on the all-flash storage system when the back-end latency is greater than or equal to the latency threshold, so as to obtain corresponding logical diagnosis data, and judge whether the logical diagnosis data meets a preset aggregation requirement.

[0167] The second stop diagnosis subunit is configured to stop the log volume fault diagnosis operation if the logical diagnosis data does not meet the aggregation requirement.

[0168] The aggregation sub-unit is configured to perform a corresponding fault repair operation on the log volume fault if the logical diagnostic data meets an aggregation requirement, and determine a fault diagnosis mechanism corresponding to the log volume fault based on a corresponding log volume fault diagnosis process of the all-flash storage system, and introduce the fault diagnosis mechanism into the fault automatic repair model.

[0169] Optionally, in an embodiment of the present application, the sending unit comprises an analysis sub-unit, a determination sub-unit and a repair sub-unit.

[0170] The analysis sub-unit is configured to analyze the fault report to obtain corresponding report analysis data, and extract a fault type, a fault influence range, a diagnosis result and a service interruption state information in the report analysis data.

[0171] The determination sub-unit is configured to determine the manual repair target data based on the fault type, the fault influence range, the diagnosis result and the service interruption state information.

[0172] The repair sub-unit is configured to determine a corresponding manual repair scheme according to the manual repair target data, and perform a manual repair operation on the fault based on the manual repair scheme.

[0173] Optionally, in an embodiment of the present application, the classification module comprises a collection unit, a standardization unit, a division unit and an extraction unit.

[0174] The collection unit is configured to collect historical fault data of the thin pool corresponding to the all-flash storage system, wherein the historical fault data of the thin pool comprises running parameter change data, fault phenomenon description information and fault category labels.

[0175] The standardization unit is configured to perform standardization processing on the historical fault data of the thin pool to obtain corresponding standard historical fault data, and construct a corresponding fault feature data set based on the standard historical fault data.

[0176] The division unit is configured to divide the fault feature data set into a training subset and a verification subset, use the training subset to train a pre-constructed fault classification initial model, and input the verification subset into the trained fault classification initial model to adjust network parameters of the fault classification initial model through the verification subset, so as to generate a target fault classification model.

[0177] The extraction unit is configured to obtain to-be-classified fault data of a plurality of thin pool faults corresponding to the all-flash storage system, extract a feature vector corresponding to the to-be-classified fault data, and input the feature vector into the target fault classification model to output fault types corresponding to the plurality of thin pool faults.

[0178] The features of the embodiments of the fault repairing device of the all-flash storage system can be referred to the related descriptions of the embodiments of the fault repairing method of the all-flash storage system, which will not be repeated here.

[0179] The embodiments of the present application also provide an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-mentioned embodiments of the fault repairing method of the all-flash storage system.

[0180] The embodiments of the present application also provide a non-volatile computer readable storage medium, which stores a computer program, and the computer program is configured to perform the steps in any of the above-mentioned embodiments of the fault repairing method of the all-flash storage system when running.

[0181] In an example embodiment, the above-mentioned non-volatile computer readable storage medium can include but is not limited to: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0182] The embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned embodiments of the fault repairing method of the all-flash storage system.

[0183] The embodiments of the present application also provide another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned embodiments of the fault repairing method of the all-flash storage system.

[0184] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0185] The above describes in detail the fault repair method, device, equipment and medium of the full-flash storage system provided by the present application. The principles and implementation modes of the present application are described by applying specific examples, and the above description of the embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that, for ordinary skilled persons in the technical field, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A fault repair method for an all-flash storage system, characterized in that, Includes the following steps: The system detects whether a fault occurs during the operation of the all-flash storage system, and if a fault is detected, determines the fault type, and calls the corresponding fault diagnosis mechanism from a pre-built automatic fault repair model according to the fault type. The fault diagnosis mechanism is used to diagnose the cause of the all-flash storage system's failure, and based on the cause of the failure, it is determined whether the all-flash storage system meets the preset repair conditions. If the preset repair conditions are met, the cause of the fault is input into the automatic fault repair model to call the fault repair mechanism corresponding to the cause of the fault, and the all-flash storage system is repaired through the fault repair mechanism; otherwise, a manual repair prompt is given. Before detecting whether a fault has occurred during the operation of the all-flash storage system, the process also includes: The various thin pool faults of the all-flash storage system are classified, and fault self-checking and repair mechanisms corresponding to different types of thin pool faults are matched. The fault self-diagnosis and repair mechanism is imported into the fault automatic repair model corresponding to the all-flash storage system, wherein the fault self-diagnosis and repair mechanism includes a fault detection mechanism, a fault diagnosis mechanism, a fault repair mechanism and a fault feedback mechanism; The step of importing the fault self-diagnosis and repair mechanism into the fault automatic repair model corresponding to the all-flash storage system includes: A thin pool fault detection operation is performed on the all-flash storage system to obtain the corresponding detection results. The fault detection mechanism is determined based on the thin pool fault detection process and imported into the fault automatic repair model. The thin pool faults include log volume faults, configuration management faults, metadata faults, deduplication faults, compression faults, and garbage collection faults. Based on the detection results, it is determined whether the all-flash storage system has the thin pool fault. In the case that the all-flash storage system has the thin pool fault, a thin pool fault diagnosis operation is performed to generate corresponding diagnostic data. Based on the thin pool fault diagnosis process, the fault diagnosis mechanism is determined and the fault diagnosis mechanism is imported into the fault automatic repair model. Based on the diagnostic data, the fault repair action corresponding to the simplified pool fault is executed, and the fault feedback information corresponding to the fault repair is generated. The fault feedback information is used to re-perform fault detection operation on the all-flash storage system. The fault repair mechanism and the fault feedback mechanism are determined according to the fault repair process and the fault feedback information generation process, and the fault repair mechanism and the fault feedback mechanism are imported into the automatic fault repair model.

2. The fault repair method for an all-flash storage system according to claim 1, characterized in that, The process of detecting whether a fault occurs in the all-flash storage system during operation, and determining the fault type when a fault is detected, and then calling the corresponding fault diagnosis mechanism from a pre-built automatic fault repair model based on the fault type, includes: Based on a preset timer polling strategy, a preset system state machine is controlled to detect whether a fault occurs in the all-flash storage system during operation; When the fault occurs during the operation of the all-flash storage system, the fault diagnosis mechanism corresponding to the fault in the fault automatic repair model is invoked, and the fault diagnosis mechanism is used to perform fault diagnosis operations to diagnose the cause of the fault in the all-flash storage system. If the all-flash storage system does not experience the aforementioned fault during operation, the system state machine is used to re-detect whether a fault has occurred during operation.

3. The fault repair method for an all-flash storage system according to claim 1, characterized in that, If the preset repair conditions are met, the cause of the fault is input into the automatic fault repair model to invoke the fault repair mechanism corresponding to the cause of the fault, and the all-flash storage system is repaired through the fault repair mechanism; otherwise, a manual repair prompt is given, including: Based on the cause of the fault, determine whether the fault meets the preset repair conditions; If the fault meets the preset repair conditions, the corresponding fault repair mechanism in the automatic fault repair model is invoked to perform the corresponding fault repair operation on the all-flash storage system through the fault repair mechanism to generate the corresponding actual fault repair information. If the fault does not meet the preset repair conditions, the corresponding business processing interruption operation is performed on the all-flash storage system, a fault report corresponding to the all-flash storage system is generated, and the fault is sent to the target user terminal so that the target user can manually repair the fault according to the fault report.

4. The fault repair method for an all-flash storage system according to claim 2, characterized in that, Also includes: The frequency of occurrence and the degree of business impact of different types of simplified pool faults in the automatic fault repair model are determined, and the different types of simplified pool faults are prioritized according to the frequency of occurrence and the degree of business impact to generate a corresponding fault priority list. The timer polling strategy is adjusted by the fault priority list, and new faults generated by the all-flash storage system during operation are processed based on the fault automatic repair model and the adjusted timer polling strategy. The detection time, diagnosis time and repair time corresponding to the new faults are recorded. Based on the detection time, diagnosis time, and repair time, the priorities of the different types of simplified pool faults and the timer polling strategy are adjusted to re-perform the corresponding fault handling operations on the all-flash storage system.

5. The fault repair method for an all-flash storage system according to claim 1, characterized in that, The step involves determining whether the all-flash storage system has a thin pool fault based on the detection results, performing a thin pool fault diagnosis operation when the all-flash storage system has a thin pool fault, generating corresponding diagnostic data, determining the fault diagnosis mechanism based on the thin pool fault diagnosis process, and importing the fault diagnosis mechanism into the automatic fault repair model, including: When performing log volume fault diagnosis on the all-flash storage system, determine the node operating mode of the all-flash storage system; If the all-flash storage system is in dual-node operation mode, then it is determined that the all-flash storage system does not have the log volume failure. If the all-flash storage system is in single-node operation mode, then the back-end latency corresponding to the all-flash storage system is detected, and it is determined whether the back-end latency is less than a preset latency threshold. When the backend latency is less than the latency threshold, stop the log volume fault diagnosis operation; When the backend latency is greater than or equal to the latency threshold, a logical diagnostic operation is performed on the all-flash storage system to obtain the corresponding logical diagnostic data, and it is determined whether the logical diagnostic data meets the preset aggregation requirements. If the logical diagnostic data does not meet the aggregation requirements, then stop the log volume fault diagnosis operation; If the logical diagnostic data meets the aggregation requirements, then the corresponding fault repair operation is performed on the log volume fault, and based on the log volume fault diagnosis process corresponding to the all-flash storage system, the fault diagnosis mechanism corresponding to the log volume fault is determined, and the fault diagnosis mechanism is imported into the automatic fault repair model.

6. The fault repair method for an all-flash storage system according to claim 3, characterized in that, The target user performs manual repair operations on the fault based on the fault report, including: The fault report is parsed to obtain the corresponding report parsing data, and the fault type, fault impact range, diagnosis results and service interruption status information in the report parsing data are extracted. Based on the fault type, the fault impact range, the diagnostic results, and the service interruption status information, determine the target data for manual repair; Based on the target data for manual repair, a corresponding manual repair plan is determined, and based on the manual repair plan, manual repair operations are performed on the fault.

7. The fault repair method for an all-flash storage system according to claim 1, characterized in that, The classification of various thin pool faults in the all-flash storage system includes: Collect historical fault data of the thin pool corresponding to the all-flash storage system, wherein the historical fault data of the thin pool includes operating parameter change data, fault phenomenon description information and fault category label; The simplified pool historical fault data is standardized to obtain corresponding standard historical fault data, and a corresponding fault feature dataset is constructed based on the standard historical fault data. The fault feature dataset is divided into a training subset and a validation subset. The training subset is used to train a pre-built initial fault classification model, and the validation subset is input into the trained initial fault classification model. The network parameters corresponding to the initial fault classification model are adjusted through the validation subset to generate a target fault classification model. Obtain the unclassified fault data of various thin pool faults corresponding to the all-flash storage system, extract the feature vectors corresponding to the unclassified fault data, and input the feature vectors into the target fault classification model to output the fault types corresponding to the various thin pool faults.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the fault repair method for the all-flash storage system as described in any one of claims 1 to 7.

Images