Troubleshooting methods and electronic devices for storage controllers

Abstract

This application discloses a fault handling method and electronic device for a storage controller, relating to the field of computer technology. The method includes: upon receiving a fault event triggered by a set of controllers, switching the cache module associated with the controller set to a first state where access is prohibited; determining the fault scenario corresponding to the controller set based on first flag information from the controller management module; if the fault scenario is a first scenario indicating the existence of a faulty mirror pair among multiple mirror pairs, sending an instruction to the cache module to prohibit the synchronization of data in the cache module to the target storage space, wherein all storage controllers included in the faulty mirror pair are in a faulty state; and switching the cache module to a second state where access is permitted, so that the storage controllers in the controller set that are in normal operating condition can use their corresponding cache modules to process data processing requests. This solves the technical problem of large data processing latency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a fault handling method and electronic device for a storage controller. Background Technology

[0002] In distributed storage systems, the storage controller, as the core component for data reading, writing, processing, and management, plays a crucial role in ensuring system stability and data security through its fault handling mechanisms. With the surge in data volume and the diversification of business needs, storage systems are increasingly evolving towards higher-end, larger-capacity, and more available solutions. In high-end storage systems, multi-controller architectures (such as quad-controller systems) are widely used to provide greater parallel processing capabilities and redundancy.

[0003] In multi-storage controller systems, especially highly available four-storage controller systems, when two storage controllers within a mirror pair fail simultaneously, existing technologies typically suspend related services until the controllers recover.

[0004] In other words, the fault handling methods for storage controllers provided in related technologies still have the technical problem of significant data processing delays due to prolonged business interruptions. Summary of the Invention

[0005] This application provides a fault handling method and electronic device for a storage controller to at least solve the problem of large data processing latency in related technologies.

[0006] This application provides a fault handling method for a storage controller, comprising: upon receiving a fault event triggered by a controller set, switching a cache module associated with the controller set to a first state where access is prohibited, wherein the storage controllers in the controller set form multiple mirror pairs, and there is a data synchronization relationship between the two storage controllers in each mirror pair; determining a fault scenario corresponding to the controller set based on a first flag information of the controller management module; if the fault scenario is a first scenario indicating that there is a faulty mirror pair among the multiple mirror pairs, sending an instruction to the cache module to indicate that data in the cache module should not be synchronized to the target storage space, wherein the storage controllers included in the faulty mirror pair are all in a faulty state; and switching the cache module to a second state where access is allowed, so that the storage controllers in the controller set that are in a normal operating state can use their corresponding cache modules to process data processing requests.

[0007] This application also provides a fault handling device for a storage controller, comprising: a first state switching unit, configured to switch a cache module associated with the controller set to a first state where access is prohibited when a fault event triggered by the controller set is obtained, wherein the storage controllers in the controller set form multiple mirror pairs, and there is a data synchronization relationship between the two storage controllers included in each mirror pair; a determining unit, configured to determine a fault scenario corresponding to the controller set based on first flag information of the controller management module; a sending unit, configured to send an instruction to the cache module to prohibit the synchronization of data in the cache module to the target storage space when the fault scenario is a first scenario indicating that there is a faulty mirror pair among the multiple mirror pairs, wherein the storage controllers included in the faulty mirror pair are all in a faulty state; and a second state switching unit, configured to switch the cache module to a second state where access is allowed, so that the storage controllers in the controller set that are in a normal operating state can use the corresponding cache module to process data processing requests.

[0008] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the fault handling method of any of the above-described memory controllers when executing the computer program.

[0009] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described fault handling methods for a storage controller.

[0010] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described memory controller fault handling methods.

[0011] In this embodiment, upon receiving a fault event triggered by a controller set, the cache module associated with the controller set is switched to a first state where access is prohibited. The storage controllers in the controller set form multiple mirror pairs, and each mirror pair contains two storage controllers with a data synchronization relationship. Based on the first flag information of the controller management module, a fault scenario corresponding to the controller set is determined. If the fault scenario is a first scenario indicating the presence of a faulty mirror pair among the multiple mirror pairs, an instruction is sent to the cache module to prohibit the synchronization of data in the cache module to the target storage space. In this case, all storage controllers in the faulty mirror pair are in a faulty state. The cache module is then switched to a second state where access is permitted, allowing the storage controllers in the controller set that are operating normally to process data processing requests using their corresponding cache modules. By promptly switching the cache module to the first state where access is prohibited, the faulty controller can be immediately prevented from further impacting the data, reducing the risk of data corruption or loss. Determining the fault scenario based on the first flag information helps to more accurately identify which mirror pairs are affected when a fault occurs, thereby avoiding unnecessary global system pauses or restarts and reducing business interruption time. Within the faulty mirror pair, instructions are sent to prevent data in the cache module from being synchronized to the target storage space, ensuring that the data of the faulty controller is not erroneously synchronized and protecting data integrity. Switching the cache module from the first state to the second state allows the normally operating controller to process data requests, thus enabling the provision of necessary data processing services even in the event of partial controller failure. This solves the technical problem of significant data processing delays caused by prolonged business interruptions in storage controller fault handling methods provided in related technologies. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a fault handling method for a storage controller provided in an embodiment of this application;

[0014] Figure 2 A schematic diagram illustrating a fault handling method for a storage controller provided in an embodiment of this application;

[0015] Figure 3 A schematic diagram illustrating another fault handling method for a storage controller provided in an embodiment of this application;

[0016] Figure 4 A flowchart of another fault handling method for a storage controller provided in an embodiment of this application;

[0017] Figure 5 This is a schematic diagram of a fault handling device for a storage controller provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0019] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0020] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] As an optional solution, the above-mentioned fault handling method for the storage controller, such as Figure 1 As shown, it includes:

[0022] S102, upon receiving a fault event triggered by the controller set, the cache module associated with the controller set is switched to the first state where access is prohibited. The storage controllers in the controller set form multiple mirror pairs, and there is a data synchronization relationship between the two storage controllers in each mirror pair.

[0023] Optionally, the aforementioned controller set refers to a collection of multiple storage controllers in a storage system that work together to handle data read / write, data synchronization, and other management tasks. The controllers in the controller set reside in the same Input / Output Group (IOGRP).

[0024] It should be noted that the above-mentioned fault events refer to hardware faults, software faults, or other abnormal states that occur in one or more storage controllers in the controller set.

[0025] Optionally, the aforementioned mirror pairs can be used, but are not limited to, to indicate a pair of storage controllers within a controller set in a storage system, establishing a data redundancy and synchronization mechanism between these pairs of storage controllers. Specifically, the two storage controllers in each mirror pair are interconnected, and through real-time or periodic data synchronization, ensure that data updates on one side are promptly reflected on the other, thereby achieving redundant data storage and enhancing data reliability and security. For example, the data stored in the memory space of a storage controller can be synchronized in real-time to the memory space of the other storage controller in the same mirror pair through the corresponding caching module.

[0026] Furthermore, the mirror pairs in the controller set can, but are not limited to, being arranged in a cyclic mirroring manner. Specifically, in a multi-controller input / output group (IOGRP), each controller forms a data synchronization mirror pair with the next controller in the IOGRP, and the last controller forms a mirror pair with the first controller, thus forming a closed-loop data synchronization network. This cyclic mirroring method not only enhances data redundancy and security but also improves system fault handling capabilities and business continuity through dynamic switching. For example, such as... Figure 2 As shown, assuming the controller set includes four storage controllers, namely controller 1, controller 2, controller 3 and controller 4, then controller 1 and controller 2 can form a mirror pair, controller 2 and controller 3 can form a mirror pair, controller 3 and controller 4 can form a mirror pair, and controller 4 and controller 1 can form a mirror pair.

[0027] It should be noted that the aforementioned first state of being prohibited from access is a temporary restricted state placed on the caching module by the system to protect data integrity. In this state, the caching module will not accept any data access requests. For example, switching the caching module to a quiescent state.

[0028] Furthermore, the step of switching the cache module associated with the controller set to the first state of being inaccessible when a fault event triggered by the controller set is obtained is executed immediately when the state machine detects a fault in the storage controller in the controller set. In order to protect the data from being affected, the state machine will put the cache module into a quiesce state.

[0029] S104, based on the first flag information of the controller management module, determine the fault scenario corresponding to the controller set.

[0030] It should be noted that the aforementioned first flag information is generated by the controller management module and is used to identify whether a controller fault exists in the current system and its nature. The aforementioned controller management module may be, but is not limited to, the cluster module of the storage cluster where the controller set resides. This cluster module is one of the core components of the storage cluster system and is mainly responsible for key tasks such as the coordination and management of all storage controllers in the cluster, resource scheduling, and fault detection and recovery.

[0031] It should be noted that the above-mentioned determination of the fault scenario corresponding to the controller set based on the first flag information of the controller management module can be used, but is not limited to, for indication. Analyzing the first flag information provided by the controller management module can help the system identify whether it is a single controller fault, multiple controller faults, or a specific mirror pair fault.

[0032] S106, in the case where the fault scenario is the first scenario used to indicate that there is a faulty mirror pair among multiple mirror pairs, an instruction is sent to the cache module to indicate that the data in the cache module should not be synchronized to the target storage space, wherein the storage controllers included in the faulty mirror pair are all in a faulty state.

[0033] Optionally, the aforementioned instruction to the cache module to prohibit the synchronization of data in the cache module to the target storage space may, but is not limited to, be completed under an acknowledgment (ACK) process.

[0034] It should be noted that the above-mentioned fault mirror pair refers to one or more mirror pairs in the controller set, in which two storage controllers fail simultaneously or successively.

[0035] Optionally, the target storage space mentioned above refers to the storage medium used in the storage system for persistently caching data, such as a hard disk or solid-state storage device.

[0036] Furthermore, in the case where the fault scenario is the first scenario used to indicate that there is a faulty mirror pair among multiple mirror pairs, the step of sending an instruction to the cache module to indicate that the data in the cache module should not be synchronized to the target storage space can be, but is not limited to, executed when the state machine determines that there is a faulty mirror pair. The purpose is to prevent the data of the fault controller from being synchronized to the target storage space, thereby avoiding data inconsistency or data loss.

[0037] S108, switch the cache module to the second state that allows access, so that the storage controllers in the controller set that are in normal operation can use the corresponding cache module to process data processing requests.

[0038] It should be noted that the aforementioned data processing requests can be, but are not limited to, read data requests, write data requests, or data management operation requests. These are business operation instructions received by the storage system during normal operation. Switching the cache module to the second accessible state can be, but is not limited to, instructing the initiation of a resume process for the cache module.

[0039] Optionally, in this embodiment, each storage controller in the controller set may, but is not limited to, be configured with a cache module and a state machine, and each state machine needs to execute the above steps separately.

[0040] It should be noted that the step of switching the cache module to the second accessible state, so that the storage controllers in the controller set that are in normal operation can use the corresponding cache module to process data processing requests, is executed after the state machine confirms that at least one storage controller in the controller set is in normal operation. The cache module switches from the inaccessible state to the accessible state to ensure that the normally operating controller can continue to process data requests and reduce business interruption.

[0041] In this embodiment, upon receiving a fault event triggered by a controller set, the cache module associated with the controller set is switched to a first state where access is prohibited. The storage controllers in the controller set form multiple mirror pairs, and each mirror pair contains two storage controllers with a data synchronization relationship. Based on the first flag information of the controller management module, a fault scenario corresponding to the controller set is determined. If the fault scenario is a first scenario indicating the presence of a faulty mirror pair among the multiple mirror pairs, an instruction is sent to the cache module to prohibit the synchronization of data in the cache module to the target storage space. In this case, all storage controllers in the faulty mirror pair are in a faulty state. The cache module is then switched to a second state where access is permitted, allowing the storage controllers in the controller set that are operating normally to process data processing requests using their corresponding cache modules. By promptly switching the cache module to the first state where access is prohibited, the faulty controller can be immediately prevented from further impacting the data, reducing the risk of data corruption or loss. Determining the fault scenario based on the first flag information helps to more accurately identify which mirror pairs are affected when a fault occurs, thereby avoiding unnecessary global system pauses or restarts and reducing business interruption time. Within the faulty mirror pair, instructions are sent to prevent data in the cache module from being synchronized to the target storage space, ensuring that the data of the faulty controller is not erroneously synchronized and protecting data integrity. Switching the cache module from the first state to the second state allows the normally operating controller to process data requests, thus enabling the provision of necessary data processing services even in the event of partial controller failure. This solves the technical problem of significant data processing delays caused by prolonged business interruptions in storage controller fault handling methods provided in related technologies.

[0042] As an optional approach, after sending an instruction to the caching module to instruct it to prevent the synchronization of data in the caching module to the target storage space, the following is also included:

[0043] Based on the first controller information obtained from the controller management module, the second controller information stored in the cache module is updated. The first controller information is used to indicate the running status of the storage controller in the current controller set, and the second controller information is used to indicate the data status of the cached data corresponding to the storage controller in the controller set. The data status is used to indicate the data integrity of the cached data of the storage controller.

[0044] The second controller information stored in the cache module is sent to the controller management module.

[0045] It should be noted that the aforementioned first controller information includes, but is not limited to, key parameters such as the health status, online / offline status, and fault recovery progress of each storage controller. It is used to reflect the operating status of the entire controller set in real time.

[0046] Optionally, the aforementioned second controller information is used to store and indicate the data status of cached data for each storage controller in the controller set. It can help the system understand whether the cached data of each controller is complete, available, or under repair.

[0047] It should be noted that the above data status includes, but is not limited to, information such as data integrity, data synchronization status, and data access permissions. It ensures that the caching module can make correct access and synchronization decisions based on the actual data situation.

[0048] Optionally, updating the second controller information stored in the cache module based on the first controller information obtained from the controller management module may include, but is not limited to, parsing the first controller information to identify the latest operating status of each stored controller in the controller set, such as online / offline status. Then, the parsed first controller information is compared with the currently stored second controller information in the cache module to identify controllers whose data status has changed, especially those controllers whose cached data may be affected. The integrity of the cached data is assessed based on the operating status of the stored controllers in the first controller information. For example, if controller A is marked as offline, the integrity of the cached data associated with A is checked, and it is determined whether it needs to be marked as incomplete. For controllers whose data status has changed, the corresponding data status flag in the second controller information is updated. For example, if controller B changes from online to faulty, and its cached data status flag was originally complete, this flag needs to be adjusted to incomplete to reflect the actual data situation.

[0049] Optionally, in this embodiment, after updating the second controller information, it may include, but is not limited to, updating the master storage controller information of the master storage controller in each of the multiple image pairs stored in the cache module.

[0050] Furthermore, when sending the second controller information stored in the cache module to the controller management module, the main storage controller information may also be sent to the controller management module at the same time, but is not limited to this.

[0051] It should be noted that the first controller information mentioned above may include, but is not limited to, two types of information: information characterizing the operating status of each controller in the controller cluster, and information characterizing the operating status of the controller within each mirror pair corresponding to the controller cluster. Correspondingly, the second controller information mentioned above may also include, but is not limited to, two types of information: information characterizing the data status of the cached data of each controller in the controller cluster, and information characterizing the data status of the cached data of the controller within each mirror pair corresponding to the controller cluster.

[0052] Optionally, the two types of information included in the first controller information above can be represented, but are not limited to, int type arrays. Specifically, an array is used to represent the running status of each controller in the controller cluster, with different running statuses corresponding to different characters, such as 1 indicating online and 0 indicating offline. Multiple arrays are used to represent the running status information of controllers within multiple mirror pairs in the controller cluster, with different running statuses corresponding to different characters, such as 1 indicating online and 0 indicating offline.

[0053] Furthermore, the presentation of the second controller information may be, but is not limited to, referencing the second controller information, and will not be elaborated further in this embodiment.

[0054] In this embodiment, based on the first controller information obtained from the controller management module, the second controller information stored in the cache module is updated. The first controller information indicates the operating status of the storage controllers in the current controller set, and the second controller information indicates the data status of the cached data corresponding to the storage controllers in the controller set. The data status indicates the data integrity of the cached data of the storage controllers. The second controller information stored in the cache module is then sent to the controller management module. By updating the second controller information in this embodiment, information synchronization between the cache module and the controller management module is ensured, reflecting the operating status of the controllers and the integrity of the cached data in real time. The updated data status enables the cache module to make more reasonable access decisions based on data integrity, such as restricting data synchronization when data is incomplete, thus avoiding potential errors caused by data synchronization.

[0055] As an optional approach, after switching the cache module to a second, accessible state so that the storage controllers in the controller set that are in normal operation can use their corresponding cache modules to process data processing requests, the following additional steps are also included:

[0056] S1, upon receiving a fault recovery event triggered by the controller set, switch the cache module to the first state.

[0057] Optionally, the fault recovery event triggered by the aforementioned controller set may, but is not limited to, be used to represent an event when one or more storage controllers in the controller set recover from a fault state to a normal operating state.

[0058] It should be noted that switching the cache module to the first state mentioned above means that when a fault recovery event is received, the cache module re-enters the protected state where access is prohibited.

[0059] S2, based on the first flag information of the controller management module, determines the fault scenario corresponding to the controller set.

[0060] S3, in the case where the fault scenario is the second scenario used to indicate that there is no faulty mirror pair among multiple mirror pairs, sends an instruction to the cache module to indicate that the data in the cache module can be synchronized to the target storage space.

[0061] It should be noted that the absence of faulty mirror pairs mentioned above can be used, but is not limited to, to indicate that there are no mirror pairs in which the included storage controllers are in a faulty state.

[0062] In this embodiment, when a fault recovery event triggered by the controller set is obtained, the cache module is switched to a first state; based on the first flag information of the controller management module, the fault scenario corresponding to the controller set is determined; when the fault scenario is a second scenario indicating that there is no faulty mirror pair among multiple mirror pairs, an instruction is sent to the cache module to indicate that the data in the cache module can be synchronized to the target storage space; and the cache module is switched to a second state that allows access.

[0063] In this embodiment, upon detecting a fault recovery event, the cache module is immediately switched to the first state. This is to reconfirm the system status and avoid prematurely activating the cache module before the fault is fully recovered, which could lead to potential data problems. When the fault scenario is determined to be the second scenario (i.e., no fault mirror pair exists), data synchronization is allowed. This avoids performing unnecessary data integrity checks under normal circumstances, improving the system's response speed and efficiency.

[0064] As an optional solution, after switching the cache module to the second state that allows access, the following is also included:

[0065] S1, based on the third controller information obtained from the controller management module, update the second controller information stored in the cache module, wherein the third controller information is used to indicate the running status of the stored controllers in the current controller set.

[0066] Optionally, the third controller information mentioned above is similar to the first controller information mentioned above; both are used to represent the real-time operating status of the controllers stored in the current controller set. However, the third controller information is obtained after the state machine receives a fault recovery event, while the first controller information is obtained after the state machine receives a fault event.

[0067] Furthermore, for the presentation method and content of the aforementioned third controller information, please refer to the explanation of the first controller information above; this will not be repeated in this embodiment.

[0068] S2 sends the second controller information stored in the cache module to the controller management module.

[0069] In this embodiment, based on the third controller information obtained from the controller management module, the second controller information stored in the cache module is updated. The third controller information indicates the operating status of the stored controllers in the current controller set. The second controller information stored in the cache module is then sent to the controller management module. By using this embodiment, after fault recovery, updating the second controller information in the cache module using the third controller information ensures that the cache module can reflect the latest operating status of the controllers in real time, providing a foundation for subsequent fault handling and data integrity management. Information exchange between the cache module and the controller management module enhances the coordination between system components, contributing to improved overall system stability and fault recovery capabilities.

[0070] As an optional approach, updating the second controller information stored in the cache module based on the third controller information obtained from the controller management module includes:

[0071] S1, obtain multiple first arrays from the third controller information and multiple second arrays from the second controller information. The values ​​of each element in each of the multiple first arrays are used to represent the running status of each storage controller in each of the multiple mirror pairs. The values ​​of each element in each of the multiple second arrays are used to represent the data status of the cached data of each storage controller in each of the multiple mirror pairs.

[0072] Optionally, the first array may be, but is not limited to, an array of type int, and the second array may also be, but is not limited to, an array of type int. In this embodiment, there is no limitation on this.

[0073] Furthermore, the first array mentioned above includes multiple elements, each element's value corresponding to the operating status of a controller within a mirror pair in the storage system, used to reflect the status changes of each mirror pair in the controller set in real time. Optionally, the second array mentioned above also includes multiple elements, each element's value corresponding to the data status of cached data of a controller within a mirror pair in the storage system, used to reflect the data status of each mirror pair in the controller set in real time.

[0074] This is used to store the historical operating status information of the controller associated with the mirror pair. Each mirror pair is associated with a second array for subsequent status updates and data synchronization decisions.

[0075] S2 compares the values ​​of elements in each of the multiple first arrays and the values ​​of elements in each of the multiple second arrays.

[0076] It should be noted that the comparison of the values ​​of elements in each of the multiple first arrays and the multiple second arrays may include, but is not limited to, comparing the i-th first array and the i-th second array in the multiple first arrays, where i is a positive integer, and the i-th first array and the i-th second array each match the i-th mirror pair in the multiple mirror pairs. In other words, when performing the above comparison steps, the first and second arrays of the same mirror pair are compared.

[0077] Furthermore, when comparing the i-th first array and the i-th second array, the comparison involves the j-th element in the i-th first array and the j-th element in the i-th second array, where j is a positive integer less than or equal to 2. The j-th element in the i-th first array and the j-th element in the i-th second array each match the same storage controller. In other words, when performing the above comparison steps, the comparison involves the elements corresponding to the same storage controller in the same mirror pair in the two arrays.

[0078] S3, update the first cache data of the first storage controller in the controller set whose data state is the first data state according to the comparison result, wherein the first data state is used to indicate that the first cache data is in an incomplete state.

[0079] It should be noted that the first cache data of the first storage controller in the controller set whose data state is the first data state is updated according to the comparison result may include, but is not limited to: when the comparison result indicates that the first element corresponding to the first storage controller in the first target array of multiple first arrays is inconsistent with the second element corresponding to the first storage controller in the second target array, the first cache data corresponding to the first storage controller is updated using the second cache data of the second storage controller in the target mirror pair.

[0080] S4, when the first cached data corresponding to the first storage controller is switched to the second data state, the element corresponding to the first storage controller in the second target array corresponding to the first storage controller in the multiple second arrays is updated, wherein the target mirror pair matching the second target array includes the first storage controller.

[0081] Optionally, the aforementioned second data state indicates that the data is in a complete, secure, or accessible state after processing. Once the state of the cached data changes from the first data state to the second data state, it means that the integrity and consistency of the data have been restored.

[0082] For example, the above content can be illustrated by examples, but is not limited to the following:

[0083] The online node settings for the cache module (i.e., the valid cache data nodes) are configured through the cluster module's online nodes. This task mainly includes two parts: four-controller iogrp granularity (using an integer type data to represent the valid controller information of the cache module in the entire iogrp) and four sets of cache mirror pair granularity in iogrp (using an integer type array of 4 to represent the valid controller information of each mirror pair in iogrp). Specifically, it includes: 1) Cache module iogrp online controller update: The cache module obtains the online controller information of each mirror pair from the cluster. If the controller represented by the 0th bit of each mirror pair is invalid, it means that there is no valid controller in the entire iogrp, and cache data loss has occurred in each mirror pair in iogrp. In this case, the online controller information in the cache module's iogrp remains unchanged; otherwise, the online controller in the cache module's iogrp is the common part of the online controller currently stored by the cluster and the cache. 2) Online controller information update for each mirror pair in the IOGRP cache module: If there are still live nodes in the cluster for the mirror pair, the online controller of the cache module stores the common part of the online controller for both the cluster and the cache (i.e., AND operation, because when the controller fails or recovers, the cluster online nodes will be updated, while the cache module needs to complete the cache data processing before updating. Therefore, there may be differences between the cluster online nodes and the cache module. The AND operation is used to ensure that the real controller storing valid cache data is obtained). If there are no live nodes in the cluster for the mirror pair (i.e., the mirror pair that caused IOGRP to enter the stall state), the cache module does not update the online node information (because the online node information in the cache module represents the node storing valid cache data. When the mirror pair nodes fail simultaneously, the valid cache data has not been synchronized and still exists in the failed node. The storage system needs to wait for the controller to recover. Therefore, the online node information of the cache module is not updated at this time).

[0084] In this embodiment, multiple first arrays from the third controller information and multiple second arrays from the second controller information are obtained. The values ​​of each element in each of the multiple first arrays represent the operating status of each storage controller in each of the multiple mirror pairs. The values ​​of the elements in each of the multiple first arrays and the values ​​of the elements in each of the multiple second arrays are compared. Based on the comparison results, the first cache data of the first storage controller in the controller set with a first data state is updated, where the first data state indicates that the first cache data is incomplete. When the first cache data corresponding to the first storage controller is switched to a second data state, the elements corresponding to the first storage controller in the second target array corresponding to the first storage controller in the multiple second arrays are updated, where the target mirror pair matching the second target array includes the first storage controller. Using this embodiment, by comparing the element values ​​in the first and second arrays, it is possible to accurately identify which controllers have incomplete cache data, which helps to perform targeted data recovery operations instead of blindly restarting the entire system. For cached data that is in an incomplete state, measures can be taken to recover it, such as using data from another controller in the mirror pair to update it, thus ensuring data integrity and consistency.

[0085] As an optional approach, updating the first cache data of the first storage controller in the controller set whose data state is the first data state based on the comparison result includes:

[0086] If the comparison result indicates that the first element corresponding to the first storage controller in the first target array of multiple first arrays is inconsistent with the second element corresponding to the first storage controller in the second target array, the first cache data corresponding to the first storage controller is updated using the second cache data of the second storage controller in the target mirror pair.

[0087] It should be noted that the above comparison results indicate that, during the fault detection process, the system uses the comparison of elements in multiple first and second target arrays to determine whether the data status has changed.

[0088] Optionally, the first element mentioned above refers to the portion of the first target array that specifically refers to the state information of the first storage controller. The second element mentioned above refers to the data state of the cached data of the first storage controller in the second target array.

[0089] In other words, in this embodiment, if the comparison result indicates that the same storage controller (i.e., the first storage controller) in the same mirror pair is in a normal operating state recorded on the cluster module side, but the cached data recorded on the cache module side is incomplete, it can be determined that the data of the normally operating first storage controller needs to be repaired to ensure the data integrity of the first storage controller. Therefore, in this case, the first cached data corresponding to the first storage controller will be updated according to the cached data of the second storage controller in the target mirror pair.

[0090] In this embodiment, when the comparison result indicates that the first element corresponding to the first storage controller in the first target array of multiple first arrays is inconsistent with the second element corresponding to the first storage controller in the second target array, the first cache data corresponding to the first storage controller is updated using the second cache data of the second storage controller in the target mirror pair. By adopting this embodiment, in the case of inconsistent element values, the incomplete cache data is updated using another normally functioning controller in the target mirror pair. This not only improves data availability but also optimizes the efficiency and accuracy of data recovery.

[0091] As an optional approach, when switching the first cached data corresponding to the first storage controller to the second data state, updating the elements corresponding to the first storage controller in the second target array of multiple second arrays includes:

[0092] When the first cached data corresponding to the first storage controller is switched to the second data state, the value of the element corresponding to the first storage controller in the second target array is modified to a value that matches the second data state.

[0093] For example, assuming the element value corresponding to the first storage controller in the second target array was originally 0, then if it is determined that the cached data of the first storage controller has been synchronized, the element value corresponding to the first storage controller in the second target array can be configured to 1.

[0094] In this embodiment, when the first cached data corresponding to the first storage controller is switched to the second data state, the value of the element corresponding to the first storage controller in the second target array is modified to a value matching the second data state. By adopting this embodiment, when cached data switches from an incomplete state to a complete state, the internal state flag of the cache module is updated. This allows the system to understand the change in data state in real time and adjust strategies promptly, such as opening data access or synchronization operations, thereby improving the system's flexibility and response speed.

[0095] As an optional approach, based on the first flag information of the controller management module, the fault scenarios corresponding to the controller set can be determined as follows:

[0096] If the first flag information is a first value, the fault scenario is determined to be the first scenario.

[0097] If the first flag information is the second value, the fault scenario is determined to be the second scenario, where the second scenario is used to indicate that there is no faulty mirror pair among the multiple mirror pairs.

[0098] For example, if the STALL flag is used to represent the first flag information mentioned above, then if STALL is true, it can be determined that the current scenario is a scenario where the data is lost due to the simultaneous failure of the controllers of the same mirror pair (i.e., the first scenario). If STALL is false, it can be determined that the current scenario is a scenario where a single controller fails or two controllers fail at intervals, without causing the loss of cached data in the same mirror pair (i.e., the second scenario).

[0099] It should be noted that the state machine can also add an internal isStalled flag in the cache module, which is also used to record fault scenarios. For specific numerical configurations, please refer to the STALL configuration; this will not be elaborated upon in this embodiment.

[0100] In this embodiment, when the first flag information is a first value, the fault scenario is determined to be a first scenario; when the first flag information is a second value, the fault scenario is determined to be a second scenario, wherein the second scenario is used to indicate that there is no faulty mirror pair among multiple mirror pairs. By using this embodiment, the specific fault scenario is determined by distinguishing different values ​​of the first flag information, which helps the system to adopt more accurate processing strategies when facing different types of faults.

[0101] As an optional approach, after determining the fault scenario as the second scenario when the first flag information is the second value, the following further steps are included:

[0102] S1, using the storage controllers in the controller set that are in normal operation, generate a mirror pair update strategy for updating the mirror pairs corresponding to the controller set.

[0103] S2, send the mirror pair update strategy to the controller management module so that the controller management module updates the mirror pairs corresponding to the controller set according to the mirror pair update strategy.

[0104] S3, switch the cache module to the second state.

[0105] It should be noted that the above mirror pair update strategy can be used, but is not limited to, to guide the storage system on how to dynamically adjust the composition of mirror pairs in the controller set under specific fault recovery scenarios in order to meet the data recovery needs and resource reallocation after controller failure.

[0106] For example, suppose the controller set includes four storage controllers: controller 1, controller 2, controller 3, and controller 4. Originally, controller 1 and controller 2 formed mirror pair 1, controller 2 and controller 3 formed mirror pair 2, controller 3 and controller 4 formed mirror pair 3, and controller 4 and controller 1 formed controller 4. However, since controller 1 is now damaged, multiple mirror pairs can be reassembled based on the following strategy, specifically, as follows: Figure 3 As shown, controller 2 and controller 3 can form a mirror pair, controller 3 and controller 4 can form a mirror pair, and controller 4 and controller 2 can form a mirror pair.

[0107] In this embodiment, a mirror pair update strategy is generated using the storage controllers in the controller set that are in normal operating state. This strategy is then sent to the controller management module, which updates the mirror pairs corresponding to the controller set according to the strategy. The cache module is then switched to a second state. By generating a mirror pair update strategy, this embodiment allows the system to more intelligently determine how to update the mirror pairs, ensuring data consistency and system stability.

[0108] As an optional solution, after fault recovery is completed, a data integrity verification mechanism can be further used to ensure the integrity and consistency of cached data, including:

[0109] Perform data integrity checks. For mirror pairs that have recovered from controller failures, use data check codes or other data verification algorithms to compare the data states before and after the failure.

[0110] If data inconsistency or integrity issues are found, immediately initiate the data repair process to restore the data using live nodes or pre-stored redundant data.

[0111] After the data repair is completed, perform the data integrity check again until it is confirmed that all data statuses are consistent and complete.

[0112] In this embodiment of the application, by adding data integrity verification and repair methods, the caching module can more accurately confirm the data status, effectively prevent data inconsistency or integrity issues, and significantly improve data security.

[0113] As an optional example, it can be, but is not limited to, by means of, such as Figure 4The following steps illustrate the troubleshooting methods for the aforementioned storage controller:

[0114] In step S402, the cluster module determines that the input / output group is in a scenario where the same mirror pair fails simultaneously.

[0115] Step S404: The cache state machine acquires fault events. Specifically, the cache module sequentially receives cluster events indicating faults in two controllers of the same mirror pair.

[0116] Step S406: The cache state machine executes a silent process, specifically: triggering the operation of the cluster event processing state machine within the cache module and initiating a business quiesce process.

[0117] Step S408: The cache state machine executes the confirmation process. Specifically, after the Quiesce process is completed, the cache module obtains the stalled flag status value from the cluster module. This flag indicates whether the cluster has determined whether a scenario has occurred in iogrp where simultaneous failures of controllers in the same mirror pair have led to data loss. If the flag is identified as true, the corresponding ack process is initiated for this scenario to synchronize information such as online nodes in the current iogrp (if a single controller failure or two controllers fail at different intervals, without causing data loss in the cache of the same mirror pair, the cluster stalled flag is false). Simultaneously, the cache module adds an internal isStalled flag to record the stall status within the module.

[0118] Furthermore, the above ack process specifically includes: adding and setting this ack task as STALL, notifying the cache module's business side via an event, and stopping the cache flushing action in the STALL state. The online node information (i.e., the cache data valid node) within the cache module's business is set through the cluster module's online nodes. This task mainly includes two parts: four-controller iogrp granularity (using an int type data to represent the valid controller information of the cache module in the entire iogrp) and four sets of cache mirror pair granularity in iogrp (using an int type array of 4 to represent the valid controller information of each mirror pair in iogrp). The cache module's iogrp online controller update includes: the cache module obtains the current online controller information of each mirror pair from the cluster. If the controller represented by the 0th bit of each mirror pair is invalid, it means there is no valid controller in the entire iogrp, and cache data loss has occurred in each mirror pair in iogrp. In this case, the online controller information in the cache module's iogrp remains unchanged; otherwise, the online controller in the cache module's iogrp is the common part of the online controller currently stored by the cluster and the cache.

[0119] The online controller information update for each mirror pair in the iogrp cache module includes the following: If there are still live nodes in the cluster for the mirror pair, the online controller of the cache module stores the common part of the online controller for both the cluster and the cache (i.e., AND operation, because when the controller fails or recovers, the cluster online nodes will be updated, while the cache module needs to complete the cache data processing before updating. Therefore, there may be differences between the cluster online nodes and the cache module. The AND operation ensures that the real controller storing valid cache data is obtained); If there are no live nodes in the cluster for the mirror pair (i.e., the mirror pair that caused iogrp to enter the stall state), the cache module does not update the online node information (because the online node information in the cache module represents the node storing valid cache data. When the mirror pair nodes fail simultaneously, the valid cache data has not been synchronized and still exists in the failed node. The storage system needs to wait for the controller to recover, so the online node information of the cache module is not updated at this time).

[0120] Then, by obtaining the cluster mirror pair controller information, the master controller (owner) information for each mirror pair in the cache module is set and transmitted to the cache module's business side. The cache module's mirror pair grouping information is updated and fed back to the cluster module to confirm that the cache module has correctly completed the handling of controller failure events and information updates.

[0121] Step S410: The cache state machine executes the recovery process. Specifically, after the Ack process is completed, the cache module restores its business processing capabilities according to the completed resume process.

[0122] Step S412: The cache state machine obtains the fault recovery event. Specifically, when the fault controller recovers, the cache module receives the controller recovery cluster event sent by the cluster.

[0123] Step S414: The cache state machine executes a silent process. Specifically, the state machine is triggered to run again. If it is detected that the cluster stalled flag has become false, the state machine is triggered to initiate the quiesce process again.

[0124] Step S416: The cache state machine executes the confirmation process and the recovery process to complete the business recovery. Specifically: After the Quiesce process ends, the state machine makes a comprehensive judgment based on the status of the cluster stalled flag and the cache module isStalled flag (the cluster flag has become false, while the internal cache flag is still true), initiates a new ack process, and completes the subsequent resume process to complete the final recovery of the business.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0126] Embodiments of this application also provide a fault handling device for a storage controller, such as... Figure 5 As shown, the fault handling device for the storage controller includes:

[0127] The first state switching unit 502 is used to switch the cache module associated with the controller set to a first state where access is prohibited when a fault event triggered by the controller set is obtained. The storage controllers in the controller set form multiple mirror pairs, and there is a data synchronization relationship between the two storage controllers in each mirror pair.

[0128] The determination unit 504 is used to determine the fault scenario corresponding to the controller set based on the first flag information of the controller management module.

[0129] The sending unit 506 is used to send an instruction to the cache module to prohibit the synchronization of data in the cache module to the target storage space when the fault scenario is a first scenario that indicates that there is a faulty mirror pair among multiple mirror pairs. The storage controllers included in the faulty mirror pair are all in a faulty state.

[0130] The second state switching unit 508 is used to switch the cache module to a second state that allows access, so that the storage controller in the controller set that is in normal operation can use the corresponding cache module to process data processing requests.

[0131] Optionally, in this embodiment, the above-mentioned device further includes: an update unit, configured to update the second controller information stored in the cache module based on the first controller information obtained from the controller management module, wherein the first controller information is used to indicate the running status of the storage controller in the current controller set, the second controller information is used to indicate the data status of the cached data corresponding to the storage controller in the controller set, and the data status is used to indicate the data integrity of the cached data of the storage controller; and a first sending unit, configured to send the second controller information stored in the cache module to the controller management module.

[0132] Optionally, in this embodiment, the above-mentioned device further includes: a third state switching unit, used to switch the cache module to a first state when a fault recovery event triggered by the controller set is obtained; a first determining unit, used to determine the fault scenario corresponding to the controller set based on the first flag information of the controller management module; a second sending unit, used to send an instruction to the cache module indicating that the data in the cache module can be synchronized to the target storage space when the fault scenario is a second scenario indicating that there is no faulty mirror pair among multiple mirror pairs; and a fourth state switching unit, used to switch the cache module to a second state that allows access.

[0133] Optionally, in this embodiment, the above-mentioned device further includes: a first updating unit, configured to update the second controller information stored in the cache module based on the third controller information obtained from the controller management module, wherein the third controller information is used to indicate the running status of the stored controllers in the current controller set; and a third sending unit, configured to send the second controller information stored in the cache module to the controller management module.

[0134] Optionally, in this embodiment, the first update unit is further configured to: obtain multiple first arrays in the third controller information and multiple second arrays in the second controller information, wherein the value of each element in each of the multiple first arrays is used to characterize the running state of each storage controller in each of the multiple mirror pairs, and the value of each element in each of the multiple second arrays is used to characterize the data state of the cached data of each storage controller in each of the multiple mirror pairs; compare the values ​​of the elements in each of the multiple first arrays and the values ​​of the elements in each of the multiple second arrays; update the first cached data of the first storage controller in the controller set whose data state is the first data state according to the comparison result, wherein the first data state is used to indicate that the first cached data is in an incomplete state; when the first cached data corresponding to the first storage controller is switched to the second data state, update the element corresponding to the first storage controller in the second target array corresponding to the first storage controller in the multiple second arrays, wherein the target mirror pair matching the second target array includes the first storage controller.

[0135] Optionally, in this embodiment, the first update unit is further configured to: update the first cache data corresponding to the first storage controller using the second cache data of the second storage controller in the target mirror pair when the comparison result indicates that the first element corresponding to the first storage controller in the first target array of multiple first arrays is inconsistent with the second element corresponding to the first storage controller in the second target array.

[0136] Optionally, in this embodiment, the first update unit is further configured to: when the first cache data corresponding to the first storage controller is switched to the second data state, modify the value of the element corresponding to the first storage controller in the second target array to a value that matches the second data state.

[0137] Optionally, in this embodiment, the determining unit is further configured to: determine the fault scenario as a first scenario when the first flag information is a first value; and determine the fault scenario as a second scenario when the first flag information is a second value, wherein the second scenario is used to indicate that there is no faulty mirror pair among the multiple mirror pairs.

[0138] Optionally, in this embodiment, the above-mentioned apparatus further includes: a generation unit, used to generate a mirror pair update strategy for updating the mirror pairs corresponding to the controller set using the storage controllers in the controller set that are in normal operation; a fourth sending unit, used to send the mirror pair update strategy to the controller management module so that the controller management module updates the mirror pairs corresponding to the controller set according to the mirror pair update strategy; and a fifth state switching unit, used to switch the cache module to the second state.

[0139] For a description of the features in the embodiment corresponding to the fault handling device of the storage controller, please refer to the relevant description of the embodiment corresponding to the fault handling method of the storage controller, which will not be repeated here.

[0140] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described embodiments of the fault handling method for a memory controller.

[0141] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the fault handling method for a storage controller when it is run.

[0142] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0143] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described storage controller fault handling method embodiments.

[0144] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the fault handling method for a storage controller.

[0145] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0146] The above provides a detailed description of a fault handling method for a storage controller provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A fault handling method for a storage controller, characterized in that, include: In the event of a fault event triggered by the controller set, the cache module associated with the controller set is switched to a first state where access is prohibited. The storage controllers in the controller set form multiple mirror pairs, and there is a data synchronization relationship between the two storage controllers in each mirror pair. Based on the first flag information of the controller management module, the fault scenario corresponding to the controller set is determined; In the case where the fault scenario is a first scenario used to indicate that there is a faulty mirror pair among the multiple mirror pairs, an instruction is sent to the cache module to indicate that the data in the cache module should not be synchronized to the target storage space, wherein the storage controllers included in the faulty mirror pair are all in a faulty state; Switch the cache module to the second state that allows access, so that the storage controllers in the controller set that are in normal operation can use the corresponding cache module to process data processing requests; Upon receiving a fault recovery event triggered by the controller set, the cache module is switched to the first state; based on the first flag information of the controller management module, the fault scenario corresponding to the controller set is determined; In the case where the failure scenario is a second scenario indicating that the failure mirror pair does not exist among the plurality of mirror pairs, an instruction is sent to the cache module to indicate that the data in the cache module can be synchronized to the target storage space; and the cache module is switched to the second state that allows access.

2. The fault handling method for the storage controller according to claim 1, characterized in that, After sending an instruction to the cache module to instruct that data in the cache module be prohibited from being synchronized to the target storage space, the method further includes: Based on the first controller information obtained from the controller management module, the second controller information stored in the cache module is updated. The first controller information is used to indicate the running status of the storage controller in the current controller set, and the second controller information is used to indicate the data status of the cached data corresponding to the storage controller in the controller set. The data status is used to indicate the data integrity of the cached data of the storage controller. The second controller information stored in the cache module is sent to the controller management module.

3. The fault handling method for the storage controller according to claim 2, characterized in that, After switching the cache module to the second state where access is permitted, the method further includes: Based on the third controller information obtained from the controller management module, the second controller information stored in the cache module is updated, wherein the third controller information is used to indicate the current running status of the stored controllers in the controller set; The second controller information stored in the cache module is sent to the controller management module.

4. The fault handling method for the storage controller according to claim 3, characterized in that, The step of updating the second controller information stored in the cache module based on the third controller information obtained from the controller management module includes: Obtain multiple first arrays from the third controller information and multiple second arrays from the second controller information, wherein the value of each element in each of the multiple first arrays is used to characterize the running status of each storage controller in each of the multiple mirror pairs, and the value of each element in each of the multiple second arrays is used to characterize the data status of the cached data of each storage controller in each of the multiple mirror pairs. Compare the values ​​of the elements in each of the plurality of first arrays with the values ​​of the elements in each of the plurality of second arrays; The first cache data of the first storage controller in the controller set with the data status of the first data status is updated according to the comparison result, wherein the first data status is used to indicate that the first cache data is in an incomplete state; When the first cached data corresponding to the first storage controller is switched to the second data state, the element corresponding to the first storage controller in the second target array corresponding to the first storage controller in the plurality of second arrays is updated, wherein the target mirror pair matching the second target array includes the first storage controller.

5. The fault handling method for the storage controller according to claim 4, characterized in that, The step of updating the first cache data of the first storage controller in the controller set to the first data state based on the comparison result includes: If the comparison result indicates that the first element corresponding to the first storage controller in the first target array of the plurality of first arrays is inconsistent with the second element corresponding to the first storage controller in the second target array, the first cache data corresponding to the first storage controller is updated using the second cache data of the second storage controller in the target mirror pair.

6. The fault handling method for the storage controller according to claim 5, characterized in that, When switching the first cached data corresponding to the first storage controller to the second data state, updating the element corresponding to the first storage controller in the second target array of the plurality of second arrays includes: When the first cached data corresponding to the first storage controller is switched to the second data state, the value of the element corresponding to the first storage controller in the second target array is modified to a value that matches the second data state.

7. The fault handling method for the storage controller according to claim 1, characterized in that, The fault scenarios corresponding to the controller set determined based on the first flag information of the controller management module include: If the first flag information is a first value, the fault scenario is determined to be the first scenario; When the first flag information is a second value, the fault scenario is determined to be a second scenario, wherein the second scenario is used to indicate that the faulty mirror pair does not exist among the plurality of mirror pairs.

8. The fault handling method for the storage controller according to claim 7, characterized in that, If the first flag information is a second value, after determining that the fault scenario is the second scenario, the method further includes: Using the storage controllers in the controller set that are in normal operating state, a mirror pair update strategy is generated for updating the mirror pairs corresponding to the controller set; The mirror pair update strategy is sent to the controller management module so that the controller management module updates the mirror pairs corresponding to the controller set according to the mirror pair update strategy. Switch the cache module to the second state.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the fault handling method for the memory controller as described in any one of claims 1 to 8 when executing the computer program.

Images