Fault processing system, method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a fault processing system, method and device, a storage medium and electronic equipment, and relates to the field of computers.The fault processing system comprises a fault obtaining module used for obtaining target fault information of a target component in the system; the fault analysis module determines a target fault type associated with the target fault information according to the component type of the target component, and one component type corresponds to one fault type; the fault recovery module is used for executing target fault recovery operations corresponding to the target fault types, and one fault type corresponds to one fault recovery operation; and the fault feedback module is used for feeding back a target processing result of the target fault reduction recovery operation to the state machine module, and the state machine module adjusts the service processing state of the storage system according to the target processing result. By adopting the scheme, the problem of low fault processing efficiency in related technologies is solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and more specifically, to a fault handling system, method and device, storage medium and electronic device. Background Art

[0002] In storage systems, failures are inevitable, especially in high-load, high-concurrency business environments. Disk failures, controller anomalies, metadata corruption and other events frequently occur, seriously threatening system stability and data security.

[0003] The current fault handling mechanism often relies on manual intervention, and the entire process from fault discovery to fault location to fault recovery is time-consuming. In addition, manual intervention may also lead to inconsistent processing and inefficiency, especially when facing complex faults, which often require multiple attempts and adjustments to find the correct recovery method.

[0004] Therefore, there is a technical problem in the prior art that the efficiency of fault handling is low. Summary of the invention

[0005] The embodiments of the present application provide a fault handling system, method and device, storage medium and electronic device to at least solve the technical problem of low fault handling efficiency in related technologies.

[0006] According to an embodiment of the present application, a fault handling system is provided, including a fault acquisition module, which is used to obtain target fault information of a target component in a system; a fault analysis module, which determines a target fault type associated with the target fault information according to a component type of the target component, wherein one component type corresponds to one fault type; a fault recovery module, which is used to perform a target fault recovery operation corresponding to the target fault type, wherein one fault type corresponds to one fault recovery operation; and a fault feedback module, which is used to feed back a target processing result of the target fault recovery operation to a state machine module, wherein the state machine module adjusts the business processing state of the storage system according to the target processing result.

[0007] According to an embodiment of the present application, a fault handling method is provided, including: receiving target fault information of a target component in a storage system; analyzing the target fault information and determining a target fault type of the target fault information; performing a target recovery operation corresponding to the target fault type according to the target fault type; and adjusting the business processing state of the storage system according to a target processing result of the target recovery operation.

[0008] According to another embodiment of the present application, a fault handling device is provided, including: a receiving unit, used to receive target fault information of a target component in a storage system; an analyzing unit, used to analyze the target fault information and determine a target fault type of the target fault information; a recovery unit, used to perform a target recovery operation corresponding to the target fault type according to the target fault type; and an adjustment unit, used to adjust the business processing state of the storage system according to a target processing result of the target recovery operation.

[0009] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps in the above method embodiment when running.

[0010] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in the above method embodiment.

[0011] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the method described in each embodiment of the present application are implemented.

[0012] Through the embodiments provided by the present application, the fault acquisition module can capture the fault information of the target component in the system in real time, and the fault analysis module can intelligently identify and determine the fault type according to the component type. This process is automated and efficient, avoiding the time-consuming manual log analysis and fault location, and shortening the response time of fault processing. The fault recovery module performs a fault recovery operation that matches the fault type, which can be based on but not limited to a pre-registered recovery strategy, and can take the most optimized recovery means for different types of faults in a targeted manner. This design reduces the blindness of fault processing and improves the efficiency and success rate of fault recovery. The fault feedback module feeds back the results of the fault recovery operation to the state machine module, and the state machine module dynamically adjusts the business processing state of the storage system according to these results. This means that the system can intelligently decide whether to continue business processing, report an alarm, or interrupt the business according to the actual situation of fault recovery, avoiding unnecessary business interruptions and enhancing the business continuity of the system. The entire fault handling process does not require manual intervention, and the system automatically completes the entire process from fault detection to recovery, greatly improving the processing efficiency. At the same time, due to the fast and intelligent fault handling, the impact of the fault on the business is reduced, and the user experience and customer satisfaction are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a structural block diagram of a fault handling system according to an embodiment of the present application;

[0014] Figure 2 is a flowchart of a fault handling method according to an embodiment of the present application;

[0015] Figure 3 It is a structural block diagram of a fault handling device according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0017] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0018] As an alternative, Figure 1 As shown, the fault handling system includes:

[0019] A fault acquisition module 102 is used to acquire target fault information of a target component in the system;

[0020] The fault analysis module 104 determines the target fault type associated with the target fault information according to the component type of the target component, wherein one component type corresponds to one fault type;

[0021] A fault recovery module 106, configured to execute a target fault recovery operation corresponding to the target fault type, wherein one fault type corresponds to one fault recovery operation;

[0022] The fault feedback module 108 is used to feed back the target processing result of the target fault recovery operation to the state machine module, wherein the state machine module adjusts the service processing state of the storage system according to the target processing result.

[0023] Optionally, in this embodiment, the target component may be, but is not limited to, any subsystem or module in the storage RAID system that may fail, including but not limited to disks, controllers, metadata management modules, configuration management modules, IO processing modules, etc.

[0024] Optionally, in this embodiment, RAID (Redundant Array of Independent Disks) is a disk array technology that combines multiple independent disk devices into a logical unit to improve data read and write performance and data redundancy protection.

[0025] Optionally, in this embodiment, the fault type refers to a fault category classified according to the type of the target component and the abnormal situation that may occur. For example, the fault type for the disk may include disk offline, disk read and write errors, etc.; the fault type for metadata management may include metadata inconsistency, metadata damage, etc.

[0026] Optionally, in this embodiment, the state machine module is the core control unit in the fault handling system, which is responsible for monitoring the operating status of the entire system and dynamically adjusting the business processing status of the system according to the received fault information and processing results to ensure the stability of the system during the fault recovery process.

[0027] Optionally, in this embodiment, the fault acquisition module is the starting point of the automatic detection and repair system, responsible for real-time monitoring of the status of each component in the storage RAID system, and once a target component failure is detected, immediately obtain detailed target failure information, including failure time, failure type, failure impact range, etc. The introduction of this module enables the system to capture the failure in the first place, avoids the delay of failure information, and improves the failure response speed.

[0028] Optionally, in this embodiment, the fault analysis module receives the target fault information transmitted by the fault acquisition module, and determines the fault type according to the component type of the target component. This process is intelligent, that is, the module can automatically identify the specific fault type according to the component type and the characteristics of the fault information. For example, if the fault information shows that there is an error in disk reading and writing, and the component type is a disk, then the fault type is determined to be a disk reading and writing fault. The accurate classification of the fault analysis module provides a clear direction for subsequent fault recovery, avoiding blind processing and waste of resources.

[0029] Optionally, in this embodiment, the fault recovery module performs corresponding fault recovery operations according to the target fault type determined by the fault analysis module. Here, one fault type corresponds to a specific fault recovery operation. For example, the recovery operation for a disk read / write fault may be disk remapping or replacement of a faulty disk, while the recovery operation for a metadata inconsistency fault may be metadata consistency verification and repair. The efficient execution of the fault recovery module ensures that the fault can be repaired in a timely manner, reduces the impact time of the fault on the business, and enhances the recovery capability of the system.

[0030] Optionally, in this embodiment, after the fault recovery operation is completed, the fault feedback module feeds back the fault processing result to the state machine module, including information such as whether the fault is successfully recovered, the time consumed for recovery, whether further manual intervention is required, etc. After receiving this information, the state machine module adjusts the business processing state of the storage system according to the processing result, such as resuming normal business processing, continuing the alarm state, or interrupting the business. This dynamic adjustment mechanism ensures the stable operation of the system during the fault recovery process, avoids unnecessary business interruptions, and at the same time improves the transparency of fault processing, which is convenient for system maintenance and user supervision.

[0031] Optionally, in this embodiment, the state machine module is the decision center of the entire fault processing system. It dynamically adjusts the business processing state of the system according to the processing results transmitted by the fault feedback module, combined with the current operating state and business needs of the system. For example, after detecting a disk failure, the state machine module can first adjust the business processing state to a degraded mode, wait for the fault recovery module to complete the disk recovery operation, and then determine whether the business processing state can be safely restored to a normal mode based on the information of the fault feedback module. The intelligent decision-making of the state machine module ensures that the system can operate in an optimal state, while reducing the uncertainty and risk in the fault recovery process.

[0032] It is understandable that through the fault acquisition module, fault analysis module, fault recovery module and fault feedback module, as well as the intelligent control of the state machine module, it is possible to achieve rapid automatic detection, accurate classification, efficient recovery and dynamic feedback of faults in the storage RAID system. The collaborative work of this series of modules significantly shortens the fault handling time, reduces the necessity of manual intervention, and enhances the stability and business continuity of the system. At the same time, through the accurate classification of fault types and the efficient execution of recovery operations, the success rate of fault recovery is also improved, and the security and integrity of data are protected.

[0033] Through the embodiments provided by the present application, the fault acquisition module can capture the fault information of the target component in the system in real time, and the fault analysis module can intelligently identify and determine the fault type according to the component type. This process is automated and efficient, avoiding the time-consuming manual log analysis and fault location, and shortening the response time of fault processing. The fault recovery module performs a fault recovery operation that matches the fault type, which can be based on but not limited to a pre-registered recovery strategy, and can take the most optimized recovery means for different types of faults in a targeted manner. This design reduces the blindness of fault processing and improves the efficiency and success rate of fault recovery. The fault feedback module feeds back the results of the fault recovery operation to the state machine module, and the state machine module dynamically adjusts the business processing state of the storage system according to these results. This means that the system can intelligently decide whether to continue business processing, report an alarm, or interrupt the business according to the actual situation of fault recovery, avoiding unnecessary business interruptions and enhancing the business continuity of the system. The entire fault handling process does not require manual intervention, and the system automatically completes the entire process from fault detection to recovery, greatly improving the processing efficiency. At the same time, due to the fast and intelligent fault handling, the impact of the fault on the business is reduced, and the user experience and customer satisfaction are improved.

[0034] As an optional solution, the system also includes a fault registration module, which is used to register corresponding registration information for components of different component types in the storage system, wherein the registration information is used to indicate the fault type and fault recovery operation associated with the fault information of the components of the different component types.

[0035] Optionally, in this embodiment, the fault registration module is part of the fault handling system, and its task is to register corresponding handling strategies for fault information of different component types in the storage system before or during the system operation. This includes the identification of fault types, the definition and association of fault recovery operations, and the setting of fault handling priorities.

[0036] Optionally, in this embodiment, the registration information refers to detailed information of fault types and fault recovery operations associated with components of different component types, including but not limited to fault triggering conditions, fault recovery steps, fault handling priorities, etc. This information provides the fault analysis module and the fault recovery module with a basis and strategy for handling faults.

[0037] Optionally, in this embodiment, the fault registration module is responsible for registering specific fault information for different types of components in the storage system (such as RAID disks, controllers, metadata management modules, etc.), and each type of fault information is associated with a specific fault type and fault recovery operation. For example, for a controller fault, the registration information may include an alarm condition for the controller to be offline, a fault diagnosis process, a recovery operation for the controller to be restarted, and a fault recovery priority, etc.

[0038] It is understandable that at the beginning of the fault handling system design, based on the long-term experience of R&D personnel and system maintenance personnel, the fault registration module pre-defines and stores various possible fault scenarios and their corresponding fault types and recovery strategies in the system. In this way, when the fault acquisition module detects the target fault information of the target component in the system, the fault analysis module can immediately compare the registration information, determine the fault type, and guide the fault recovery module to perform the corresponding fault recovery operation.

[0039] It should be noted that the fault registration module may have, but is not limited to, the following functions:

[0040] (1) Matching fault types with recovery actions: Define possible fault types for each component type and accurately match them with corresponding fault recovery actions to ensure that the system can take the most appropriate action when a fault occurs.

[0041] (2) Fault handling priority setting: Fault handling priorities are set based on the potential impact of faults on business continuity and data security, so that the system can prioritize handling of the most urgent and critical faults to minimize business interruption time and data loss risks.

[0042] (3) Dynamic update capability: The fault registration module should have the ability to dynamically update registration information to adapt to the upgrade, expansion and environmental changes of the storage RAID system. This includes the ability to add new component types, fault types and recovery operations in a timely manner, as well as adjust fault handling priorities.

[0043] Through the embodiments provided in the present application, when a fault occurs, the fault type can be quickly located and predefined fault recovery operations can be executed, which significantly reduces the response time and recovery time of fault handling and improves the accuracy and efficiency of fault handling.

[0044] As an optional solution, the fault registration module includes a first fault registration submodule, a second fault registration submodule, a third fault registration submodule and a fourth fault registration submodule.

[0045] The first fault registration submodule is used to register corresponding first registration information for a component of the metadata type in the storage system, wherein the first registration information is used to indicate a first fault type and a first fault recovery operation associated with the fault information of the component of the metadata type;

[0046] The second fault registration submodule is used to register corresponding second registration information for the configuration management type component in the storage system, wherein the second registration information is used to indicate a second fault type and a second fault recovery operation associated with the fault information of the configuration management type component;

[0047] The third fault registration submodule is used to register corresponding third registration information for the data input and output type component in the storage system, wherein the third registration information is used to indicate the third fault type and the third fault recovery operation associated with the fault information of the data input and output type component;

[0048] The fourth fault registration submodule is used to register corresponding fourth registration information for the background task type component in the storage system, wherein the fourth registration information is used to indicate the fourth fault type and fourth fault recovery operation associated with the fault information of the background task type component.

[0049] Optionally, in this embodiment, the first fault registration submodule is used to register the fault type and fault recovery operation related to metadata type components in the storage system. Metadata components are usually responsible for processing key data such as disk mapping, file system information, RAID configuration, etc., and their failures have a direct impact on the stability and data security of the storage system.

[0050] Optionally, in this embodiment, the second fault registration submodule is used to register fault types and fault recovery operations related to configuration management for components of configuration management type, such as system RAID configuration, disk management, operation status monitoring, etc. Failure of configuration management components may affect the availability and performance of the storage system.

[0051] Optionally, in this embodiment, the third fault registration submodule is for data input and output type components, such as read and write operations, data transmission, etc. of the storage system, and registers the fault type and fault recovery operation related to data input and output. Failure of such components may cause data read and write errors or delays, affecting the data access performance of the storage system.

[0052] Optionally, in this embodiment, the fourth fault registration submodule is a component responsible for background task types, including but not limited to RAID reconstruction, system initialization tasks, etc., and registers fault types and fault recovery operations related to background tasks. Failure of background task components may affect the data reconstruction efficiency and system initialization process of the storage system.

[0053] Optionally, in this embodiment, the first fault registration submodule registers the first registration information for the metadata component in the storage system. This registration information includes the first fault type and the first fault recovery operation associated with the fault information of the metadata component. For example, when the metadata of the storage system is inconsistent, the first registration information will guide the fault analysis module to identify the fault type and instruct the fault recovery module to perform a specific first fault recovery operation, such as metadata consistency check and repair.

[0054] Optionally, in this embodiment, the second fault registration submodule registers second registration information related to the configuration management type component. This includes the second fault type and the second fault recovery operation associated with the fault information of the configuration management component. For example, when the system detects that the RAID configuration is abnormal, the second registration information will guide the system to take measures such as automatic configuration recovery and state correction to prevent system instability or performance degradation caused by configuration problems.

[0055] Optionally, in this embodiment, the third fault registration submodule registers third registration information for data input / output type components, which is used to indicate the fault type and recovery operation of the data input / output component. For example, when the storage system encounters a read / write error, the third registration information can quickly guide the system to take measures such as data path switching, error data isolation or retransmission to ensure the continuity and correctness of data access.

[0056] Optionally, in this embodiment, the fourth fault registration submodule focuses on the component of the background task type and registers the fourth registration information to indicate the fault type and recovery operation of the background task component. For example, during the RAID reconstruction process, if the disk data is damaged or the reconstruction fails, the fourth registration information will guide the system to automatically perform operations such as data repair, disk replacement or reconstruction task restart to ensure the smooth progress of the RAID reconstruction task and the integrity of data redundancy.

[0057] Through the embodiments provided in this application, it is possible to achieve refined management and rapid response to various types of faults in the storage RAID system. Each submodule specifies fault identification, diagnosis and recovery strategies for specific types of components, which not only speeds up fault handling and improves the success rate of fault recovery, but also reduces dependence on manual intervention and enhances the automation and intelligence level of the system.

[0058] As an optional solution, the failure recovery module includes a first failure recovery module, a second failure recovery module, a third failure recovery module and a fourth failure recovery module.

[0059] The first fault recovery module is used to perform the first fault recovery operation corresponding to the first fault type; the second fault recovery module is used to perform the second fault recovery operation corresponding to the first fault type; the third fault recovery module is used to perform the third fault recovery operation corresponding to the first fault type; and the fourth fault recovery module is used to perform the fourth fault recovery operation corresponding to the first fault type.

[0060] Optionally, in this embodiment, the first failure recovery module performs a predefined first failure recovery operation for the first failure type registered by the first fault registration submodule to restore the normal operation of the metadata component. Failures of the metadata component may include but are not limited to metadata inconsistency, metadata damage, etc., and the recovery operations for these failures generally involve metadata verification, data recovery, or system restart, etc.

[0061] Optionally, in this embodiment, the second fault recovery module faces the second fault type registered by the second fault registration submodule, and the second fault recovery module performs the corresponding second fault recovery operation. The failure of the configuration management component may include loss of configuration information, abnormal configuration status, etc., and the recovery operation may require measures such as reconfiguring RAID, repairing configuration files, or adjusting system status.

[0062] Optionally, in this embodiment, the third fault recovery module performs a predefined third fault recovery operation for the third fault type registered by the third fault registration submodule to restore the normal function of the data input and output component. The failure of such components may be manifested as data read and write errors, transmission delays, etc., and the recovery operation may include data path switching, error data isolation, data retransmission, etc.

[0063] Optionally, in this embodiment, the fourth fault recovery module performs a corresponding fourth fault recovery operation for the fourth fault type registered by the fourth fault registration submodule to solve the fault problem of the background task component. Failures of background tasks such as RAID reconstruction and data migration may require operations such as data repair, disk replacement, and task restart to recover.

[0064] Through the embodiments provided by the present application, by introducing four fault recovery submodules, the fault handling system of the present invention can quickly identify the fault type when a fault occurs and perform corresponding effective recovery operations. This not only shortens the response time of fault handling and improves the success rate of fault recovery, but also reduces the dependence on manual intervention and enhances the automation and intelligence level of the system.

[0065] As an optional solution, the fault receiving module includes a monitoring submodule, and the monitoring submodule is used to obtain fault information generated by the component in the storage system in real time when monitoring that a fault occurs in the component in the storage system.

[0066] Optionally, in this embodiment, the monitoring submodule monitors the system status in real time during the operation of the storage system, and once a component failure is detected, immediately obtains detailed fault information, including the type of component where the failure occurred, the nature and severity of the failure, etc., to ensure that the fault information can be quickly collected and transmitted to other modules of the fault handling system.

[0067] Optionally, in this embodiment, the monitoring submodule may implement real-time monitoring of the storage system status in a variety of ways, including but not limited to system log analysis, performance indicator monitoring, hardware status detection, etc. Once a fault signal (such as disk read / write error, controller offline, etc.) is detected, the monitoring submodule will immediately start the fault information collection mechanism to obtain detailed fault information, including the time when the fault occurred, the specific type of the faulty component, the preliminary cause analysis of the fault, etc. This information is then passed to the fault analysis module, which decides whether to start the fault recovery process and what kind of recovery operation to take based on the fault type and the preset processing strategy.

[0068] Through the embodiments provided by this application, based on the real-time monitoring and fault information acquisition capabilities of the monitoring submodule, the response speed of fault handling is significantly accelerated, and the time window from fault occurrence to fault recovery is shortened. This design not only directly solves the technical problems of long response time and untimely acquisition of fault information in the traditional fault handling mechanism, but also improves the automation level of fault handling.

[0069] As an optional solution, the system further includes a fault prediction module, which is used to predict the fault type and the failure recovery operation of components of different component types in the storage system to obtain prediction information, and the fault registration module registers corresponding registration information for components of different component types in the storage system according to the prediction information;

[0070] The fault prediction model includes a fault prediction model, wherein the fault prediction model is a model obtained by learning and training using historical faults generated by components of different component types in the storage system, and historical fault types and historical failure recovery operations corresponding to the historical faults.

[0071] Optionally, in this embodiment, the fault prediction module uses historical fault data and machine learning technology to predict the types of faults and recovery operations that may occur in different components of the storage system. This prediction capability not only helps to take measures in advance to avoid the occurrence of faults, but also can quickly start the recovery process when a fault is about to occur, reducing the impact of the fault on the business.

[0072] Optionally, in this embodiment, the fault prediction model builds an intelligent model that can predict future faults and recovery operations by learning historical fault data of different component types in the storage system, including historical fault types and historical failure recovery operations. This model is trained based on a large number of fault cases and can predict potential fault risks and their response strategies based on the current system status and environmental changes.

[0073] Optionally, in this embodiment, the historical fault data of the storage system is analyzed through the learning and training of the fault prediction model to extract the patterns and rules of fault occurrence, as well as the historical fault types and the effectiveness of historical fault reduction and recovery operations. Based on these historical data, the fault prediction model can predict the potential fault risks in the current or future system state and give corresponding fault reduction and recovery operation suggestions.

[0074] In the specific implementation, the fault prediction module passes the prediction information to the fault registration module. This information includes the predicted fault type and the recommended fault recovery operation. The fault registration module will register the corresponding registration information for the components of different component types in the storage system according to the prediction information. This means that even before the fault actually occurs, the system already has a strategy and preparation to deal with the fault, so that it can achieve rapid response and processing of the fault.

[0075] The training and optimization of the fault prediction model is the key to realizing the function of the fault prediction module. By collecting the historical fault data of different components during the operation of the storage system, including the specific time of the fault, the fault type, the scope of the fault, the fault reduction and recovery operations taken and their effects, the fault prediction model can continuously learn and adjust to improve its accuracy in predicting fault types and recommending recovery operations.

[0076] It should be noted that the training of the fault prediction model is an ongoing process, which requires the continuous collection of new historical fault data and the use of these data to optimize and adjust the model to improve the accuracy of the prediction and the effectiveness of the recovery strategy. At the same time, the fault prediction module should have the ability to monitor the system status and environmental changes in real time, so as to adjust the prediction results according to the current situation and ensure the matching degree between the prediction information and the actual situation.

[0077] Through the embodiments provided by this application, by introducing a fault prediction module, the fault handling system is provided with active prevention and intelligent prediction functions, which greatly improves the stability and business continuity of the system. The fault prediction module can identify potential fault risks in advance, so that the system can start the fault recovery process before or at the early stage of the fault, reducing the impact time of the fault on the business and avoiding the risk of data loss and business interruption.

[0078] As an alternative, Figure 2 As shown, a fault handling method includes:

[0079] S202, receiving target failure information of a target component in a storage system;

[0080] S204, analyzing the target fault information and determining a target fault type of the target fault information;

[0081] S206, according to the target fault type, executing a target recovery operation corresponding to the target fault type;

[0082] S208: Adjust the service processing state of the storage system according to the target processing result of the target recovery operation.

[0083] Optionally, in this embodiment, the target component refers to a specific component or module in the storage system that triggers the fault, such as a disk, a controller, a metadata management module, etc. The failure of the target component may cause problems such as performance degradation, service interruption, or data loss of the storage system, so it is very important to handle the failure of the target component in a timely and effective manner.

[0084] Optionally, in this embodiment, the target fault information is the fault information detected and reported by the target component during operation, including key information such as the time when the fault occurred, component status, and fault type speculation. This information is the basis for fault analysis and recovery operations, ensuring that the fault handling system can accurately identify and respond to faults.

[0085] Optionally, in this embodiment, the target fault type is a specific fault type determined by the fault analysis module after analyzing the target fault information, such as metadata consistency fault, configuration management exception, IO process fault, etc. Accurate determination of the target fault type is a prerequisite for selecting an appropriate recovery strategy.

[0086] Optionally, in this embodiment, the target recovery operation is a failure recovery process that matches the target failure type, which is intended to eliminate the impact of the failure on the system and restore the normal operation of the component. The target recovery operation may include specific operations such as data repair, configuration recovery, and system restart.

[0087] Optionally, in this embodiment, the service processing state refers to the ability and state of the storage system to process service requests, such as normal operation, partial restriction, complete stop, etc. The ultimate goal of the fault handling method is to adjust and optimize the service processing state to ensure service continuity and data security.

[0088] Optionally, in this embodiment, the operating status of each component in the storage system is monitored in real time, and once a target component is detected to have a fault, the target fault information reported by it is immediately received to provide raw data for subsequent fault analysis and processing.

[0089] After receiving the target fault information, it is analyzed and diagnosed in detail to determine the specific type of fault reflected by the target fault information. This process may involve the analysis of fault information, the comparison of historical data, the identification of fault modes, etc., to ensure that the judgment of the fault type is accurate.

[0090] Once the target fault type is determined, the failure recovery module will select and execute the corresponding target recovery operation based on the fault type. This operation may require calling a predefined failure recovery strategy, or taking the most appropriate recovery measures based on the recommendations of the fault prediction model to restore the normal operation of the storage system or target component as soon as possible.

[0091] After the target recovery operation is completed, the fault feedback module will collect the target processing results, evaluate the effectiveness of the fault recovery and the adjustment of the business processing status. If the fault is successfully recovered, the business processing status will be adjusted to normal operation or partially restricted to ensure business continuity; if the fault cannot be automatically recovered or the recovery fails, the business processing status may be adjusted to a completely stopped state to avoid further deterioration of the fault, and the system administrator will be notified to intervene for manual processing.

[0092] Through the embodiments provided by this application, the efficiency and success rate of storage system fault handling are significantly improved by real-time reception of target component fault information, accurate analysis of fault types, rapid execution of target recovery operations, and dynamic adjustment of business processing status according to recovery results. This method not only shortens the time window from fault discovery to fault recovery, but also reduces the need for manual intervention through intelligent judgment and automated operation, thereby achieving the technical effect of improving the efficiency of fault handling.

[0093] As an optional solution, the above fault handling method is applied to the automatic detection and repair system of the storage RAID system, an automatic repair mechanism and framework is established, and the fault repair strategy is imported into the automatic detection and framework according to the maintenance summary and R&D design. Therefore, when a fault occurs, the system can automatically detect, diagnose and repair the fault, and completely package the manual intervention link, the repair plan formulation environment and the repair plan execution link into the system automatic recovery. The greater benefit of this method is not to reduce the cost of manual intervention, but to improve the system's business continuity capabilities and system fault recovery capabilities.

[0094] Optionally, the storage RAID system automatic detection and repair system is based on the storage system and provides a unified and abstract state machine for the RAID business module. It mainly includes a framework state machine, a fault receiving module, a fault diagnosis module, a fault recovery module, and a fault feedback module. Among them, the framework state machine is mainly responsible for the interaction with the business module and the promotion and flow of fault processing. The fault receiving module is mainly responsible for receiving the fault information pushed by the RAID business module. The fault diagnosis module is mainly responsible for analyzing and processing the fault. The fault recovery module is mainly responsible for recovering the fault. The fault feedback module is mainly responsible for feeding back the results to the RAID business module. The RAID business module performs the next step according to the feedback solution. The framework provides interfaces for registering the fault receiving module, fault diagnosis module, fault recovery module, and fault feedback module, and the RAID business module completes the registration.

[0095] Optionally, the storage RAID system mainly includes background tasks such as initialization and reconstruction, configuration management, task scheduling, IO processing, metadata management and other sub-components, and detailed fault handling classification is performed based on component classification.

[0096] Metadata failures: RAID metadata mainly includes metadata management of background tasks such as initialization and reconstruction. If metadata failures mean serious failures, such as abnormal reconstruction metadata, the storage system will have to stop the service. Here we register common metadata failures and related diagnosis and recovery, including initialization metadata abnormality, reconstruction metadata abnormality, and asynchrony between controllers.

[0097] Configuration management failure: Configuration management mainly involves RAID disk management, operation status management, member status management, and RAID basic status management. Configuration management failures are mainly RAID state machine abnormalities, such as internal status field check abnormalities, status stuck, etc. Here, each type of configuration management failure and related diagnosis and recovery need to be registered.

[0098] IO process failure: mainly due to abnormal status check during the IO flow process, IO cannot return or returns abnormally, etc.

[0099] Initialization, reconstruction, and other background task failures: The main failures are the same as metadata failures. Other failures include task scheduling, interaction with IO modules, and progress display.

[0100] Optionally, the RAID fault type is classified, and each fault is registered according to fault detection, fault diagnosis, fault recovery, and fault handling result feedback. Taking RAID reconstruction metadata anomaly as an example, the RAID business module first registers the alarm. When a fault occurs, the alarm can be captured by the fault automatic detection and diagnosis mechanism. The fault diagnosis mechanism is also registered. It should be noted that the fault diagnosis mechanism is provided by the development and maintenance personnel of the business module, that is, how to diagnose and locate this kind of fault. This location step is imported according to the specifications required by the automatic diagnosis and recovery system framework. The same logic is used to register fault recovery and fault result feedback. The architecture is running in a state. After a fault alarm is captured, the diagnostic interface of the business module is called for diagnosis. After diagnosis, it is found that the cause is the mismatch of metadata between controllers. The analysis result is given. According to the analysis result, the state machine enters the next recovery action. If there is a response recovery action, it is executed. Otherwise, the feedback cannot execute the recovery action, and the alarm is retained, which reminds users that the fault needs technical personnel to intervene and solve.

[0101] It should be noted that the business module reports the alarm to the state machine, the state machine checks the alarm registered by the business, diagnoses the diagnostic interface registered by the business module, and performs recovery actions. If there is no problem in each step, it will be executed step by step according to the plan until the system is restored by default. Otherwise, it will enter the default process and no processing will be performed.

[0102] To further illustrate, the storage RAID system detects fault reporting by timer polling; once a fault occurs, because the fault has been registered to the automatic detection system, the automatic detection system calls the diagnosis corresponding to the fault and performs recovery actions according to the diagnosis results; if the diagnosis results determine that the recovery action can be performed, the recovery action interface is called for recovery. After the recovery is completed, if the system recovery is determined, if the system has completed the recovery, the RAID module is notified to continue business processing. If the system has not completed the recovery, the RAID module is notified that the fault has not been eliminated, please continue to report the alarm or interrupt business processing; if the diagnosis results determine that the problem cannot be solved by automatic recovery means, the recovery action is not executed, and the alarm is continued to be reported or business processing is interrupted. This scenario is a special fault scenario, waiting for manual intervention to solve; after the system completes processing a fault, the state machine starts to process the next fault in a polling manner;

[0103] Through the embodiments provided by the present application, a RAID fault automatic repair system is established by combining the service characteristics of RAID, the fault repair method and the automatic repair mechanism framework, that is, the RAID fault types are classified, and each fault is registered according to fault detection, fault diagnosis, fault recovery and fault handling result feedback. After the system is running, the fault is automatically handled according to the injected fault diagnosis and repair means, so that the fault can be repaired in the fastest and most effective way, and the continuity of the business is guaranteed to the greatest extent.

[0104] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0105] In this embodiment, a fault handling device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0106] Figure 3 is a structural block diagram of a fault handling device according to an embodiment of the present application, such as Figure 3 As shown, the device comprises:

[0107] A receiving unit 302, configured to receive target fault information of a target component in the storage system;

[0108] An analyzing unit 304 is used to analyze the target fault information and determine a target fault type of the target fault information;

[0109] A recovery unit 306, configured to perform a target recovery operation corresponding to the target fault type according to the target fault type;

[0110] The adjusting unit 308 is configured to adjust the service processing state of the storage system according to the target processing result of the target recovery operation.

[0111] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0113] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0114] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0115] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0116] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0117] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0118] An embodiment of the present application further provides a computer program product, comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program product, and when the computer program is executed by a processor, the steps of the method described in each embodiment of the present application are implemented.

[0119] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0120] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0121] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A fault handling system, characterized in that: include: A fault acquisition module is used to obtain target fault information of a target component in the system; A fault analysis module, which determines a target fault type associated with the target fault information according to a component type of the target component, wherein one component type corresponds to one fault type; A fault recovery module, configured to execute a target fault recovery operation corresponding to the target fault type, wherein one fault type corresponds to one fault recovery operation; A fault feedback module is used to feed back a target processing result of the target fault recovery operation to a state machine module, wherein the state machine module adjusts the business processing state of the storage system according to the target processing result.

2. The system according to claim 1, characterized in that The system also includes a fault registration module, which is used to register corresponding registration information for components of different component types in the storage system, wherein the registration information is used to indicate the fault type and fault recovery operation associated with the fault information of the components of the different component types.

3. The system according to claim 2, characterized in that The fault registration module includes a first fault registration submodule, a second fault registration submodule, a third fault registration submodule and a fourth fault registration submodule. The first fault registration submodule is used to register corresponding first registration information for a component of the metadata type in the storage system, wherein the first registration information is used to indicate a first fault type and a first fault recovery operation associated with the fault information of the component of the metadata type; The second fault registration submodule is used to register corresponding second registration information for the configuration management type component in the storage system, wherein the second registration information is used to indicate a second fault type and a second fault recovery operation associated with the fault information of the configuration management type component; The third fault registration submodule is used to register corresponding third registration information for the data input and output type component in the storage system, wherein the third registration information is used to indicate the third fault type and the third fault recovery operation associated with the fault information of the data input and output type component; The fourth fault registration submodule is used to register corresponding fourth registration information for the background task type component in the storage system, wherein the fourth registration information is used to indicate the fourth fault type and fourth fault recovery operation associated with the fault information of the background task type component.

4. The system according to claim 3, characterized in that The failure recovery module includes a first failure recovery module, a second failure recovery module, a third failure recovery module and a fourth failure recovery module. The first failure recovery module is used to perform the first failure recovery operation corresponding to the first fault type; The second failure recovery module is used to perform the second failure recovery operation corresponding to the first fault type; The third failure recovery module is used to perform the third failure recovery operation corresponding to the first fault type; The fourth fault recovery module is used to perform the fourth fault recovery operation corresponding to the first fault type.

5. The system according to any one of claims 1 to 4, characterized in that: The fault receiving module includes a monitoring submodule, and the monitoring submodule is used to obtain the fault information generated by the component in the storage system in real time when monitoring that the component in the storage system has a fault.

6. The system according to any one of claims 1 to 4, characterized in that: The system further includes a fault prediction module, which is used to predict the fault type and the fault recovery operation of the components of different component types in the storage system to obtain prediction information, and the fault registration module registers corresponding registration information for the components of different component types in the storage system according to the prediction information; The fault prediction model includes a fault prediction model, wherein the fault prediction model is a model obtained by learning and training using historical faults generated by components of different component types in the storage system, and historical fault types and historical failure recovery operations corresponding to the historical faults.

7. A fault handling method, characterized in that: include: receiving target failure information of a target component in the storage system; Analyzing the target fault information and determining a target fault type of the target fault information; According to the target fault type, executing a target recovery operation corresponding to the target fault type; The service processing state of the storage system is adjusted according to the target processing result of the target recovery operation.

8. A fault handling device, characterized in that: include: A receiving unit, configured to receive target fault information of a target component in the storage system; An analyzing unit, configured to analyze the target fault information and determine a target fault type of the target fault information; A recovery unit, configured to perform a target recovery operation corresponding to the target fault type according to the target fault type; The adjustment unit is used to adjust the business processing state of the storage system according to the target processing result of the target recovery operation.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in claim 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to claim 7 are implemented.

Images