Memory fault processing method, device, controller, system, medium and product

Abstract

The invention discloses a memory fault processing method and device, a controller, a system, a medium and a product, and relates to the field of fault processing, when a target memory has an IO overtime fault, the memory controller controls to close an IO retry function of the target memory; according to the method and the device, the first target requests of the target memory are received, the first target requests of the target memory are stopped, the first target requests of the current target memory are directly stopped, the IO requests which are not processed in the current target memory can be rapidly processed, and then an error processing thread can be called to carry out fault processing. By closing the IO retry function of the target memory, the time consumed by IO repeated retry is avoided, the IO request which is not processed completely is compressed for timeout fault detection, and the time consumed by retry is avoided, so that an error processing thread can be quickly awakened, the fault processing duration is greatly shortened, and the influence caused by overlong-time request blockage is avoided.

Description

Technical Field

[0001] This invention relates to the field of fault handling, and in particular to a method, apparatus, controller, system, medium, and product for handling memory faults. Background Technology

[0002] With the continuous development of technology, users have increasingly higher requirements for storage. However, during use, storage inevitably experiences hardware failures due to factors such as hardware aging, component failure, and firmware defects. One typical failure is I / O (Input / Output) timeout blocking. To handle storage I / O timeout failures, the Linux system introduced the scsi_eh thread (error handling thread) to process I / O requests with I / O timeout failures. Before the scsi_eh thread is triggered, the storage controller is set to the SHOST_RECOVERY state (host recovery state). In this state, all I / O requests sent to the storage controller are blocked until the scsi_eh thread completes its processing, thus preventing the continued transmission of erroneous I / O. However, in existing technologies, before triggering the scsi_eh thread, requests are repeatedly retried through IO retries. This repeated retrying operation consumes a lot of time, causing the scsi_eh thread to fail to trigger in a short period of time. This greatly increases the fault handling time of the entire IO timeout fault. When there are a large number of storage devices under the storage controller, the excessively long fault handling time will cause a large number of IO requests to be blocked, thereby causing delays in the entire storage system, or even paralyzing the entire storage system, which will seriously affect user services. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a memory fault handling method, apparatus, controller, system, medium, and product. By disabling the I / O retry function of the target memory, the time consumed by repeated I / O retries is avoided. The time consumed by timeout fault detection and retrying of unprocessed I / O requests is reduced, allowing the error handling thread to be quickly awakened, significantly shortening the fault handling time and avoiding the impact of prolonged request blocking on the storage system. The specific solution is as follows:

[0004] In a first aspect, this application discloses a memory fault handling method applied to a memory controller, comprising:

[0005] If an I / O timeout fault is detected in the target memory, control the target memory to disable the I / O retry function;

[0006] Initiate an abort operation for each first target request in the current target memory, and determine the abort operation result for each first target request; wherein, the first target request is an unprocessed IO request in the current target memory, and the abort operation result includes abort success or abort failure;

[0007] A preset error handling thread is invoked and the I / O timeout fault of the target memory is handled based on the second target request; wherein, the second target request is the first target request whose abort operation result is abort failure.

[0008] Optionally, controlling the target memory to disable the I / O retry function includes:

[0009] The target parameter variable of the target memory is set to zero to control the target memory to disable the IO retry function;

[0010] The target parameter variable is a parameter variable that characterizes the upper limit of the number of I / O retries for the target memory.

[0011] Optional, also includes:

[0012] Identify the initiator of the third target request; wherein, the third target request is the first target request whose abort operation result is successful.

[0013] Generate a request response failure result corresponding to the third target request, and transmit the request response failure result to the request initiator.

[0014] Optionally, after determining the abort operation result corresponding to each of the first target requests, the method further includes:

[0015] The first target request, whose abortion operation result is determined to be abortion failure, is used to obtain the second target request;

[0016] Call the preset function to add the second target request to the preset queue;

[0017] Accordingly, a preset error handling thread is invoked and, based on the second target request, the IO timeout fault of the target memory is handled, including:

[0018] If the abort operation results corresponding to each of the first target requests have been obtained, then a preset error handling thread is invoked to process the second target requests in the preset queue, so as to locate the target memory and handle the IO timeout fault of the located target memory.

[0019] Optionally, handling the I / O timeout fault of the located target memory includes:

[0020] A preset restart and reset operation is performed to handle the I / O timeout fault of the located target memory;

[0021] The preset restart and reset operation includes a restart and reset operation for the target memory, a reset operation for the target channel bus, and a reset operation for the target adapter; the target channel bus is the channel bus between the memory controller and the target memory, and the target adapter is the adapter between the memory controller and the target memory.

[0022] Optionally, after detecting an I / O timeout fault in the target memory, before processing the I / O timeout fault of the target memory, the method further includes:

[0023] Change the state of the storage controller from the first state to the second state;

[0024] Accordingly, after handling the I / O timeout fault of the target memory, the method further includes:

[0025] Change the state of the storage controller from the second state to the first state;

[0026] The first state indicates that the storage controller is currently allowed to receive new IO requests, and the second state indicates that the storage controller is currently refused to receive any new IO requests.

[0027] Optionally, after handling the I / O timeout fault of the target memory, the method further includes:

[0028] The target memory is isolated by prohibiting arbitrary I / O operations and is marked as offline.

[0029] Optionally, after marking the target memory as offline, the method further includes:

[0030] If the preset conditions are met, a preset control command is sent to the target memory to control the target memory to attempt to perform an I / O operation;

[0031] During the process of the target memory attempting to perform an I / O operation, the fault status of the target memory is monitored, and a determination is made based on the fault status whether the target memory should be isolated again;

[0032] The preset conditions include the target memory's continuous isolation duration reaching a preset duration threshold and / or the detection of maintenance records for the target memory.

[0033] Optionally, monitoring the fault status of the target memory and determining whether to isolate the target memory again based on the fault status includes:

[0034] Configure a fault count value for the target memory; the initial value of the fault count value is zero;

[0035] If an I / O timeout fault is detected in the target memory, the target memory is isolated again, and the fault count value is updated.

[0036] If the current fault count value is greater than a preset threshold, the target memory will be permanently isolated.

[0037] Secondly, this application discloses a memory fault handling device applied to a memory controller, comprising:

[0038] The function shutdown unit controls the target memory to disable the IO retry function if it detects an IO timeout fault in the target memory.

[0039] The abort processing unit is used to initiate an abort operation for each first target request in the current target memory and determine the abort operation result corresponding to each first target request; wherein, the first target request is an unprocessed IO request in the current target memory, and the abort operation result includes abort success or abort failure;

[0040] An error handling unit is used to invoke a preset error handling thread and, based on a second target request, process the I / O timeout fault of the target memory; wherein, the second target request is the first target request whose abort operation result is abort failure.

[0041] Thirdly, this application discloses a storage controller, which includes a processing unit and a storage unit; wherein the storage unit is used to store a computer program, which is loaded and executed by the processing unit to implement the memory fault handling method as described above.

[0042] Fourthly, this application discloses a storage system, including a storage controller as described above and a plurality of memories connected to the storage controller.

[0043] Fifthly, this application discloses a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the memory fault handling method described above.

[0044] Sixthly, this application discloses a computer program product, including a computer program / instructions, which, when executed by a processor, implements the aforementioned memory fault handling method.

[0045] This invention discloses a memory fault handling method. When an I / O timeout fault is detected in the target memory, the memory controller disables the I / O retry function of the target memory. It also aborts the first target requests in the target memory, no longer waiting for I / O timeout faults to occur, but directly aborting each of the first target requests in the current target memory. This allows unprocessed I / O requests in the current target memory to be processed quickly, and then the error handling thread can be invoked for fault handling. By disabling the I / O retry function of the target memory, the time consumed by repeated I / O retries is avoided, and the time consumed by timeout fault detection and retries for unprocessed I / O requests is reduced. This allows the error handling thread to be quickly awakened, greatly shortening the fault handling time and avoiding the impact of prolonged request blocking.

[0046] The present invention also provides a memory fault handling apparatus, a memory controller, a memory system, a computer-readable storage medium, and a computer program product, which have the same beneficial effects as the memory fault handling method described above. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 A flowchart illustrating a memory fault handling method provided by the present invention;

[0049] Figure 2 A flowchart illustrating another memory fault handling method provided by the present invention;

[0050] Figure 3 This is a schematic diagram illustrating the abort processing of the current I / O request in a memory fault handling method provided by the present invention;

[0051] Figure 4 A schematic diagram illustrating the state change control of the memory controller in a memory fault handling method provided by the present invention;

[0052] Figure 5 This is a schematic diagram illustrating the method for handling memory faults provided by the present invention, which controls the target memory to disable the I / O retry function.

[0053] Figure 6This is a schematic diagram illustrating the abort processing of other first target requests in a memory fault handling method provided by the present invention;

[0054] Figure 7 A schematic diagram illustrating the invocation of a preset error handling thread in a memory fault handling method provided by the present invention;

[0055] Figure 8 This is a schematic diagram illustrating the isolation of the target memory in a memory fault handling method provided by the present invention;

[0056] Figure 9 A schematic diagram of the structure of a memory fault handling device provided by the present invention;

[0057] Figure 10 This is a schematic diagram of the structure of a storage controller provided by the present invention. Detailed Implementation

[0058] The core of this invention is to provide a memory fault handling method, apparatus, controller, system, medium, and product. By disabling the IO retry function of the target memory, the time consumed by repeated IO retries is avoided. The time consumed by timeout fault detection and retry of unprocessed IO requests is compressed, so that the error handling thread can be quickly awakened, which greatly shortens the fault handling time and avoids the impact of excessively long request blocking on the storage system.

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] See Figure 1 As shown, Figure 1 This is a flowchart illustrating a memory fault handling method provided by the present invention; in a first aspect, this application discloses a memory fault handling method applied to a memory controller, comprising:

[0061] S11: If an I / O timeout fault is detected in the target memory, control the target memory to disable the I / O retry function;

[0062] Understandably, storage systems typically employ a storage controller to manage data efficiently. When data storage or access is required, the storage controller sends an I / O request to the memory to perform the corresponding data operation. During this I / O operation, I / O timeout failures may occur. An I / O timeout failure refers to a situation where the storage controller does not receive a response from the memory within a preset time or fails to complete the data transfer, ultimately determining the operation as failed due to timeout.

[0063] To address I / O timeout failures during memory use, the I / O processing flow in the storage controller continuously monitors the memory's response to the I / O request and / or whether the target data corresponding to the I / O request has been completely transmitted after issuing the I / O request. If no response is received from the memory within a preset time or the target data corresponding to the I / O request is not completely transmitted, the storage controller determines that the memory corresponding to the I / O request is the target memory experiencing an I / O timeout failure and triggers subsequent memory fault handling methods. The memory corresponding to the I / O request refers to the memory that receives and processes the I / O request. This application does not specify the specific value or implementation method of the preset time for triggering the I / O timeout failure; generally, the default preset time for I / O requests is 30 seconds. In practical applications, the Linux system can adjust the preset time through ` / sys / block / sdX / device / timeout`.

[0064] Furthermore, upon detecting an I / O timeout fault in the target memory, to ensure the scsi_eh thread can be quickly triggered, the storage controller immediately disables the I / O retry function of the target memory, limiting its I / O retry behavior. I / O retry refers to automatically re-initiating the I / O request when it fails, attempting to re-execute the corresponding I / O operation. Repeated I / O retries consume a significant amount of time and can prevent the scsi_eh thread from being woken up. Therefore, in this application, once an I / O timeout fault is detected in the target memory, the I / O retry function is immediately disabled, thus limiting the time consumed by repeated I / O retries. The specific method for disabling the I / O retry function is not specifically limited here.

[0065] S12: Initiate the abort operation for each first target request in the current target memory, and determine the abort operation result corresponding to each first target request; wherein, the first target request is the unprocessed IO request in the current target memory, and the abort operation result includes abort success or abort failure;

[0066] It's easy to understand that when an I / O timeout failure is detected in the target memory, the storage controller first attempts to cancel the I / O request through an abort operation. The abort operation proactively terminates the I / O request, thus preventing resources in the target memory from being occupied for an extended period. Simultaneously, to improve processing efficiency, after issuing an I / O request, the storage controller continuously receives user requests and issues the next I / O request to the memory. Therefore, a memory typically has a certain number of unprocessed I / O requests during normal operation. Thus, when an I / O timeout failure occurs in the target memory, the storage controller not only immediately performs an abort operation on the currently processing I / O request that caused the timeout, but also directly initiates corresponding abort operations on other unprocessed I / O requests, further saving the time consumed by the initial I / O timeout failure and I / O retries.

[0067] For each first target request, after executing the abort operation, the abort may succeed (successful abort) or fail (failed abort). This application does not specifically limit the specific implementation of the abort operation. For example, when the storage controller's kernel uses a Linux system, the `scsi_abort_command()` function can be executed asynchronously through the software work queue in the storage controller's kernel to send an ABORTTASK SCSI management command to the target memory, thereby aborting the IO requests that are being executed or queued by the target memory. This application does not specifically limit the specific method of obtaining the first target request. It can be achieved by directly reading the pending IO or the queue of pending requests from the target memory. The first target request refers to all IO requests that the storage controller has submitted to the target memory, that the target memory is processing, and that have not yet completed processing. Specifically, this includes the current IO request that the target memory is executing, which is the one that caused the IO timeout failure, and the IO requests waiting to be processed in the queue of pending requests from the target memory.

[0068] S13: Invoke the preset error handling thread and handle the IO timeout fault of the target memory based on the second target request; wherein, the second target request is the first target request whose abort operation result is abort failure.

[0069] Understandably, if abort fails, it means that the abort operation did not successfully abort the first target request. This could be because the first target request is still being processed or is still in the processing queue of the target memory. It could also be that the first target request has been aborted, but the resources occupied by the first target request have not been released or there are other abnormal states caused by the first target request. In this case, it is necessary to further call the preset error handling thread to perform further fault handling on the target memory based on the second target request.

[0070] This application does not specifically limit the specific type, implementation method, and calling mechanism of the preset error handling thread. Generally speaking, the preset error handling thread can only be awakened after all the first target requests to the target memory have been aborted and processed. Specifically, the preset error handling thread can be the scsi_eh thread. The scsi_eh thread is an error recovery mechanism in the Linux kernel used to handle SCSI (Small Computer System Interface) device I / O errors. When a SCSI command fails, times out, or the device becomes unresponsive, the kernel starts the scsi_eh thread to attempt to recover the device. Therefore, memory I / O timeout faults can be handled by integrating the scsi_eh thread into the storage controller kernel. Specifically, a separate scsi_eh thread is set up in the storage controller kernel, and the scsi_eh thread can be called by executing the entry function scsi_error_handler.

[0071] It should be noted that this application does not impose any special limitations on the specific types and implementation methods of the storage devices and storage controllers. The storage devices can be disks such as mechanical hard drives, solid-state drives, FC (Fibre Channel) external storage, or other types of storage devices. The storage controller can be a SAS (Serial Attached SCSI) card, a RAID (Redundant Array of Independent Disks) card, an HBA (Host Bus Adapter) card, etc. Storage controllers are used in the storage system to expand interfaces. As the host of the storage system, the storage controller can simultaneously connect multiple storage devices, thus realizing a storage system. This application does not impose any special limitations on the specific implementation method of the storage controller kernel, including but not limited to a Linux kernel implemented based on the Linux operating system.

[0072] This invention achieves fault handling for the target memory by adjusting the calling mechanism of the preset error handling thread. When an I / O timeout fault occurs in a memory connected to the storage controller, an abort operation is performed on the current I / O request that caused the I / O timeout, and an abort operation is also performed on all first target requests in the faulty target memory. Simultaneously, the I / O retry function of the faulty target memory is immediately disabled to avoid repeated retries that consume time. After these steps, all first target requests in the faulty target memory are quickly processed, and then the preset error handling thread is awakened for processing. By automatically disabling the I / O retry function of the faulty target memory, it is ensured that the first target requests in the target memory can return a failed response result in about 30 seconds, avoiding the impact of prolonged I / O blocking on upper-layer services. By disabling the I / O retry function and immediately aborting all first target requests, the fault handling time is shortened, allowing the preset error handling thread to be quickly awakened and perform fault handling, thus improving the overall efficiency of fault handling.

[0073] As an optional embodiment, controlling the target memory to disable the I / O retry function includes:

[0074] Set the target parameter variable of the current target memory to zero to control the target memory to disable the IO retry function;

[0075] Among them, the target parameter variable is a parameter variable that represents the upper limit of the number of I / O retries for the target memory.

[0076] It's easy to understand that, for centralized management, the kernel of the storage controller stores target parameter variables for all memory devices connected to the controller. If a memory device needs to perform I / O retries, the number of I / O retries is controlled according to the value assigned to the target parameter variable. Therefore, when it's necessary to disable the I / O retries function for a memory device, this can be achieved by setting the target parameter variable to 0; I / O retries will only be performed zero times, meaning no I / O retries are allowed. This application does not specifically limit the specific storage location or implementation method of the target parameter variable.

[0077] Specifically, disabling the IO retry function can be achieved directly by assigning values ​​to the target parameter variable. This is intuitive, effective, and easy to implement. The entire operation process is very simple, requiring only the direct modification of the target parameter variable's value. This approach has low implementation costs and facilitates a rapid response from the storage controller to disable the IO retry function of the target memory.

[0078] As an optional embodiment, it also includes:

[0079] Identify the initiator of the third target request; wherein, the third target request is the first target request whose operation result is successful abort.

[0080] Generate a request response failure result corresponding to the third target request, and transmit the request response failure result to the request initiator.

[0081] Understandably, if an abort succeeds, it means the corresponding first target request has been forcibly stopped, resources have been released, and it will no longer occupy the target memory's resources. This third target request will not leave any abnormal states such as resource occupation or resource leakage, and will not affect the subsequent operation of the target memory. Therefore, it is only necessary to return the request failure result of the third target request to the request initiator, informing the request initiator that the IO request was not successfully executed. Considering that the storage controller may process requests from different users simultaneously in actual applications, it is best to first determine the request initiator of the third target request before returning the request failure result to the corresponding request initiator.

[0082] This application does not impose any special restrictions on the specific type and implementation method of the request initiator at the upper layer of the storage controller, nor on the specific implementation method of the request response failure result. It is sufficient to inform the request initiator of the operation result of the third target request operation failure. Generally speaking, the storage controller will connect with the upper layer business system. The business system refers to the software system designed to meet specific business needs. Users initiate operations by operating the business system. The business system converts the user's data read and write requirements into read and write requests to the storage system based on the user's operation and sends the read and write requests to the storage controller. The storage controller receives the read and write requests from the business system and then distributes the read and write requests to each storage device. After the storage device responds to the storage controller, it executes the corresponding data transmission operation. The storage controller then returns the execution result of the read and write request to the business system to inform it whether the read and write request was executed successfully or failed, thereby completing the entire requirement processing flow.

[0083] Specifically, after performing an abort operation on each first target request, the failure result of the successfully aborted third target request can be directly fed back to the request initiator to clarify the processing result of the third target request, reduce the user's waiting time, facilitate the user to take corresponding remedial measures in a timely manner according to their needs, and optimize the user experience.

[0084] As an optional embodiment, after determining the abort operation result corresponding to each first target request, the method further includes:

[0085] The first target request is determined to be an abort failure, in order to obtain the second target request;

[0086] Call the preset function to add the second target request to the preset queue;

[0087] Accordingly, a pre-defined error handling thread is invoked, and based on the second target request, the I / O timeout fault of the target memory is handled, including:

[0088] If the abort operation results corresponding to each first target request have been obtained, the preset error handling thread is invoked to process the second target requests in the preset queue, so as to locate the target memory and handle the IO timeout fault of the located target memory.

[0089] It is easy to understand that, in order to facilitate the preset error handling thread to quickly handle memory faults based on the second target request, when performing an abort operation on the first target request, the failed second target request is directly added to the preset queue after the abort operation of any first target request is completed. This generates a preset queue that the subsequent preset error handling thread needs to process. The preset queue is used to store the failed second target requests, so as to facilitate unified processing by the subsequent preset error handling thread and achieve unified scheduling of the second target requests. This application does not make any special restrictions on the specific type and implementation method of the preset queue. Specifically, it can be implemented using eh_cmd_q (EventHandler Command Queue) in the Linux system.

[0090] Once all first-target requests have been aborted and processed, the preset error handling thread can be invoked. "All first-target requests have been aborted and processed" means that all first-target requests have been added to the preset queue as second-target requests or aborted as third-target requests and returned a failure response. This application does not specifically limit the specific type and implementation of the preset function; it can be implemented using functions such as `scsi_eh_scmd_add()`. Since the preset error handling thread exists independently in the memory controller kernel, after invoking it, the preset error handling thread will locate the faulty target memory through the second-target requests and then process the target memory device itself.

[0091] Specifically, a preset queue for storing second target requests is generated to facilitate the unified processing of second target requests by subsequent preset error handling threads. All second target requests are temporarily stored in the form of a queue to avoid the loss of second target requests and data corruption, thus ensuring the orderliness and reliability of the fault handling process.

[0092] As an optional embodiment, handling the I / O timeout fault of the located target memory includes:

[0093] Perform a preset restart and reset operation to handle the I / O timeout fault of the located target memory;

[0094] The preset restart and reset operations include restarting and resetting the target memory, resetting the target channel bus, and resetting the target adapter; the target channel bus is the channel bus between the memory controller and the target memory, and the target adapter is the adapter between the memory controller and the target memory.

[0095] Understandably, the preset error handling thread can specifically perform a restart and reset of the target memory and / or the communication connection between the target memory and the storage controller. Generally, the preset error handling thread will attempt to handle the faulty target memory using a progressively escalating operation strategy in the following order: First, control the target memory to restart to achieve a soft reset; second, control the target memory to reset to achieve a hard reset; third, reset the target channel bus to which the target memory belongs; fourth, reset the target adapter to which the target memory belongs. After each reset and restart operation, the storage controller will attempt to execute the second target request to determine whether the target memory can handle the second target request normally. If it can, the fault handling of the target memory is considered successful; if not, the next reset and restart operation will continue until all reset and restart operations are completed. If the second target request still cannot be handled normally, the fault handling of the target memory is considered to have failed, and it is best to prompt the operator to intervene manually through an alarm module or other means.

[0096] This application does not impose any specific limitations on the specific implementation methods of each reset and restart operation. Restarting the target memory can be achieved by sending a START_STOP_UNIT command, and resetting the target memory can be achieved by sending a DEVICE_RESET command. Resetting the target channel bus can also be achieved by resetting the server device of the target memory (e.g., a SCSI target). This application does not impose any specific limitations on the specific type and implementation method of the target communication bus and target adapter. The target communication bus can be a SCSI BUS (channel), and the target adapter can be an HBA adapter, etc.

[0097] Specifically, this embodiment introduces an error handling mechanism with a preset error handling thread. Its handling of IO timeout faults is a layered and progressive fault handling and recovery process, which can maximize the recovery of the target memory when the target memory fails, and realize the fault handling of the target memory.

[0098] As an optional embodiment, after detecting an I / O timeout fault in the target memory, before processing the I / O timeout fault of the target memory, the method further includes:

[0099] Change the state of the storage controller from the first state to the second state;

[0100] Accordingly, after handling the I / O timeout fault of the target memory, the process also includes:

[0101] Change the state of the storage controller from the second state to the first state;

[0102] The first state indicates that the current storage controller allows new IO requests, while the second state indicates that the current storage controller refuses to accept any new IO requests.

[0103] It's easy to understand that when the target memory experiences an I / O timeout failure, this failure could also be caused by the storage controller. In this case, if the storage controller's I / O operations are not blocked in time, erroneous I / O requests may continue to be issued. As these erroneous I / O requests accumulate, the preset error handling thread may not be able to complete its processing. Therefore, before the storage controller calls the preset error handling thread to handle the target memory failure, the storage controller's state needs to be changed to the second state. Specifically, this can be done when the second target request is added to the preset queue. The storage controller's state is then restored to the first state after the preset error handling thread has completed the target memory failure handling.

[0104] When the storage controller is in the second state, all new I / O requests are blocked. It will not receive new read / write requests from the business system, nor will it issue I / O requests to any memory. This application does not specifically limit the specific types and implementation methods of the first and second states. Specifically, the second state can be the SHOST_RECOVERY state. After the preset error handling thread completes the fault handling of the target memory, it clears the SHOST_RECOVERY state of the storage controller to restore it to the first state, resuming the issuance of new I / O requests to other memories besides the target memory.

[0105] Specifically, since the IO retry function of the target memory is disabled and the system no longer waits for the timeout faults of each first target request, the storage controller will be in the second state for a very short time after an IO timeout fault occurs in the target memory, approximately less than 1 second. Therefore, the IO requests of other memories besides the target memory will not be affected, reducing the impact of fault handling on the entire storage system and avoiding the paralysis of the storage system.

[0106] See Figure 2 As shown, Figure 2 A flowchart illustrating another memory fault handling method provided by the present invention; as an optional embodiment, after handling the IO timeout fault of the target memory, the method further includes:

[0107] S14: Disable arbitrary I / O operations on the target memory to isolate the target memory and mark it as offline.

[0108] Understandably, if the target memory continues to function normally and receive new I / O requests after a fault is successfully handled by the preset error handling thread, then the entire fault handling mechanism will repeat after another I / O timeout fault occurs, causing continuous disruption to the entire storage system's operations. Therefore, in this application, after the preset error handling thread completes its processing, regardless of whether the processing result is success or failure, the faulty target memory will be isolated, prohibiting it from performing any I / O operations. This application does not specify the specific method for isolating the target memory; it can be achieved by calling the `scsi_eh_offline_sdevs()` function to mark the target memory as offline (SDEV_OFFLINE), etc.

[0109] Specifically, after the preset error handling thread processes the error, the faulty target memory is automatically isolated, thereby avoiding the target memory from having a continuous impact on the business of the entire storage system, preventing the target memory from repeatedly triggering the fault handling process, and ensuring the normal operation of the entire storage system.

[0110] As one specific embodiment, see Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the abort processing of the current I / O request in a memory fault handling method provided by the present invention; see also Figure 4 As shown, Figure 4 This is a schematic diagram illustrating the state change control of the memory controller in a memory fault handling method provided by the present invention; see also Figure 5 As shown, Figure 5 This is a schematic diagram illustrating the method for handling memory faults provided by the present invention, specifically the method for controlling the target memory to disable the I / O retry function; see also Figure 6 As shown, Figure 6 This is a schematic diagram illustrating the abort processing of other first target requests in a memory fault handling method provided by the present invention; see also Figure 7 As shown, Figure 7 This is a schematic diagram illustrating the invocation of a preset error handling thread in a memory fault handling method provided by the present invention; see also Figure 8 As shown, Figure 8This is a schematic diagram of isolating the target memory in a memory fault handling method provided by the present invention; taking the memory as a disk, the storage controller is connected to three disks: disk 1, disk 2 and disk 3. Taking the scsi_eh thread as an example, the business system sends read and write requests and other business I / O to the storage controller, and then the disk I / O between the storage controller and the disks realizes data reading and writing.

[0111] The storage controller sequentially issues three I / O requests: IO1, IO2, and IO3. If request IO1 from disk 1 times out, the fault handling process is triggered, first performing an abort operation on request IO1, such as... Figure 3 As shown. If the abort operation for request IO1 fails, request IO1 is added to the eh_cmd_q queue of the scsi_eh thread in the storage controller; then the storage controller's state is set to SHOST_RECOVERY, after which any IO requests sent to the storage controller will be blocked, as shown. Figure 4 As shown. Simultaneously, immediately disable the I / O retry function for disk 1, controlling disk 1 to remain in noretry (no retries) state, as shown. Figure 5 As shown. Then immediately perform an abort operation on all remaining first target requests on disk 1 (including requests IO2 and IO3), as follows. Figure 6 As shown. After the above steps, all pending I / O on disk 1 will be quickly processed, and then the scsi_eh thread will be woken up to handle them, as follows. Figure 7 As shown, if requests IO1, IO2, and IO3 all fail to abort, they are all added to the eh_cmd_q queue as secondary target requests. Finally, after the scsi_eh thread finishes processing, regardless of whether the processing result is successful or not, disk 1 will be automatically isolated, marked as SDEV_OFFLINE, and the storage controller's SHOST_RECOVERY state will be cleared. Figure 8 As shown, the I / O between the storage controller and disks 2 and 3 then returned to normal, and it will no longer be affected by the faulty disk 1.

[0112] As an optional embodiment, after marking the target memory as offline, the method further includes:

[0113] If the preset conditions are met, a preset control command is sent to the target memory to control the target memory to attempt to perform an I / O operation;

[0114] During the process of the target memory attempting to perform I / O operations, the fault status of the target memory is monitored, and a decision is made based on the fault status whether to isolate the target memory again.

[0115] The preset conditions include the target memory's continuous isolation time reaching a preset time threshold and / or the detection of maintenance records for the target memory.

[0116] It is easy to understand that after isolating the target memory, a recovery process for the isolated memory can be further configured according to the actual situation. This involves controlling the target memory to attempt I / O operations to determine if I / O timeout faults still occur. The target memory can be restored after a period of isolation, or an alarm signal can be sent to the user after isolation. Upon receiving the alarm signal, the user can manually maintain and repair the target memory, and then restore it after manual repair. This application does not impose specific limitations on the preset conditions for triggering the recovery process. There are also multiple implementation methods for the specific recovery operations of the target memory, which are not specifically limited here. The I / O operations attempted can be normal read / write requests from the business system issued by the storage controller, or specific I / O requests used for testing. If the target memory can perform I / O operations normally, it can be discontinued and return to normal operation, executing I / O requests issued by the storage controller. If the target memory cannot perform I / O operations normally, it should be isolated again.

[0117] Specifically, by setting up a recovery process, the target memory that has failed can continue to work normally after the cause of the failure has been completely eliminated. This reduces the impact of the target memory on the storage system while ensuring the normal operation of each memory as much as possible.

[0118] As an optional embodiment, monitoring the fault status of the target memory and determining whether to isolate the target memory again based on the fault status includes:

[0119] Configure a fault count value for the target memory; the initial value of the fault count value is zero.

[0120] If an I / O timeout fault is detected in the current target memory, the target memory will be isolated again and the fault count value will be updated.

[0121] If the current fault count is greater than the preset threshold, the target memory will be permanently isolated.

[0122] Understandably, if the target memory fails to perform I / O operations normally, it will be isolated again. Simultaneously, a fault count can be configured for the target memory. Each time an I / O timeout occurs, the fault count is incremented. If the fault count exceeds a preset threshold, it indicates that the target memory will continue to fail repeatedly. In this case, the target memory can be permanently isolated, and its normal operation will not be restored. This application does not specifically limit the specific implementation of the fault count and preset threshold; the fault count can be configured in the kernel of the storage controller. Other methods can also be used to determine whether permanent isolation of the target memory is necessary. This application does not specifically limit this method and is not limited to using a fault count. Furthermore, a time limit can be added; for example, if the fault count exceeds a preset threshold within a preset time period, the fault count is reset to zero every preset time period.

[0123] Specifically, the number of times the target memory is recovered is counted by configuring a fault count value. The count of the fault count value is used to determine the repeated failure of the target memory. In the case of repeated failure of the target memory, it is permanently isolated to avoid its impact on the normal operation of the entire storage system.

[0124] See Figure 9 As shown, Figure 9 This invention provides a schematic diagram of the structure of a memory fault handling device; secondly, this application discloses a memory fault handling device applied to a memory controller, comprising:

[0125] If the function shutdown unit 11 detects an I / O timeout fault in the target memory, it controls the target memory to disable the I / O retry function.

[0126] The abort processing unit 12 is used to initiate abort operations on each first target request in the current target memory and determine the abort operation result corresponding to each first target request; wherein, the first target request is an unprocessed IO request in the current target memory, and the abort operation result includes abort success or abort failure.

[0127] Error handling unit 13 is used to call a preset error handling thread and process the IO timeout fault of the target memory based on the second target request; wherein, the second target request is the first target request whose abort operation result is abort failure.

[0128] As an optional embodiment, the function shutdown unit 11 includes:

[0129] The variable assignment unit is used to assign the target parameter variable of the target memory to zero so as to control the target memory to disable the IO retry function;

[0130] The target parameter variable is a parameter variable that characterizes the upper limit of the number of I / O retries for the target memory.

[0131] As an optional embodiment, it also includes:

[0132] The initiator determination unit is used to determine the initiator of the third target request; wherein the third target request is the first target request whose abort operation result is successful abort.

[0133] The successful request processing unit is used to generate a request response failure result corresponding to the third target request and transmit the request response failure result to the request initiator.

[0134] As an optional embodiment, it also includes:

[0135] The second target request determination unit is used to determine the first target request whose abort operation result is abort failure, so as to obtain the second target request;

[0136] A preset queue generation unit is used to call a preset function to add the second target request to the preset queue;

[0137] Accordingly, the error handling unit 13 includes:

[0138] An error handling subunit is used to call a preset error handling thread to process the second target request in the preset queue if the abort operation results corresponding to each of the first target requests have been obtained, so as to locate the target memory and handle the IO timeout fault of the located target memory.

[0139] As an optional embodiment, the error handling subunit includes:

[0140] The restart and reset unit is used to perform a preset restart and reset operation to handle the IO timeout fault of the located target memory;

[0141] The preset restart and reset operation includes a restart and reset operation for the target memory, a reset operation for the target channel bus, and a reset operation for the target adapter; the target channel bus is the channel bus between the memory controller and the target memory, and the target adapter is the adapter between the memory controller and the target memory.

[0142] As an optional embodiment, it also includes:

[0143] The first state change unit is used to change the state of the storage controller from the first state to the second state after detecting an IO timeout fault in the target memory and before processing the IO timeout fault of the target memory.

[0144] Correspondingly, it also includes:

[0145] The second state change unit is used to change the state of the storage controller from the second state to the first state after processing the IO timeout fault of the target memory; wherein, the first state indicates that the storage controller is currently allowed to receive new IO requests, and the second state indicates that the storage controller is currently refused to receive any new IO requests.

[0146] As an optional embodiment, it also includes:

[0147] An isolation unit is used to handle the IO timeout fault of the target memory, prohibit the target memory from performing any IO operation, so as to isolate the target memory and mark the target memory as offline.

[0148] As an optional embodiment, it also includes:

[0149] The recovery unit is used to send a preset control command to the target memory after marking it as offline, if the current preset conditions are met, to control the target memory to attempt to perform IO operations.

[0150] The repeated isolation unit is used to monitor the fault status of the target memory during the process of the target memory attempting to perform an I / O operation, and determine whether to isolate the target memory again based on the fault status;

[0151] The preset conditions include the target memory's continuous isolation duration reaching a preset duration threshold and / or the detection of maintenance records for the target memory.

[0152] As an optional embodiment, the repeating isolation unit includes:

[0153] A configuration unit is configured to configure a fault count value for the target memory; the initial value of the fault count value is zero.

[0154] The counting unit is used to isolate the target memory again and update the fault count value if an IO timeout fault is detected in the target memory.

[0155] A permanent isolation unit is used to permanently isolate the target memory if the current fault count value is greater than a preset threshold.

[0156] For a description of the memory fault handling device provided by the present invention, please refer to the embodiments of the memory fault handling method described above; the present invention will not be repeated here.

[0157] See Figure 10 As shown, Figure 10 This is a schematic diagram of the structure of a storage controller provided by the present invention. In a third aspect, this application discloses a storage controller, which includes a processing unit 21 and a storage unit 22; wherein the storage unit 22 is used to store a computer program 222, which is loaded and executed by the processing unit 21 to implement the memory fault handling method as described above.

[0158] Furthermore, embodiments of this application also disclose a storage controller. Figure 10 This is a structural diagram of a storage controller according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0159] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the storage controller; the communication interface 24 can create a data transmission channel between the storage controller and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0160] Furthermore, the storage unit 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon can include computer programs 222, etc., and the storage method can be temporary storage or permanent storage. The computer program 222 may include, in addition to a computer program capable of performing the memory fault handling method executed by the storage controller as disclosed in any of the foregoing embodiments, a computer program capable of performing other specific tasks.

[0161] For an introduction to the memory controller provided by this invention, please refer to the embodiments of the memory fault handling method described above; the invention will not be repeated here.

[0162] Fourthly, this application discloses a storage system, including a storage controller as described above and a plurality of memories connected to the storage controller.

[0163] For an introduction to the storage system provided by this invention, please refer to the embodiments of the memory fault handling method described above; the invention will not be repeated here.

[0164] Fifthly, this application discloses a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the memory fault handling method described above.

[0165] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application. Specifically, the computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, and portable hard drives, or any type of media or device suitable for storing instructions or data, etc., and this application does not make any special limitations here.

[0166] For an introduction to the computer-readable storage medium provided by the present invention, please refer to the embodiments of the memory fault handling method described above; the present invention will not be repeated here.

[0167] Sixthly, this application discloses a computer program product, including a computer program / instructions, which, when executed by a processor, implements the aforementioned memory fault handling method.

[0168] For an introduction to the computer program product provided by this invention, please refer to the embodiments of the memory fault handling method described above; the invention will not be repeated here.

[0169] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0170] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0171] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0172] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0173] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A memory failure handling method, characterized by, The application is applied to a storage controller, comprising: If an IO timeout fault of a target storage is monitored, the target storage is controlled to close an IO retry function; An abort operation of each first target request in the target storage is started, and an abort operation result corresponding to each first target request is determined; the first target request is an unprocessed IO request in the target storage, and the abort operation result comprises abort success or abort failure; A preset error processing thread is called, and the IO timeout fault of the target storage is processed based on a second target request; the second target request is the first target request with abort failure.

2. The memory fault handling method of claim 1, wherein, The target storage is controlled to close the IO retry function, comprising: A target parameter variable of the target storage is assigned to zero to control the target storage to close the IO retry function; The target parameter variable is a parameter variable representing an upper limit of the IO retry times of the target storage.

3. The memory fault handling method of claim 1, wherein, Further comprising: A request initiator of a third target request is determined; the third target request is the first target request with abort success; A request response failure result corresponding to the third target request is generated, and the request response failure result is transmitted to the request initiator.

4. The memory fault handling method of claim 1, wherein, After the abort operation result corresponding to each first target request is determined, further comprising: The first target request with abort failure is determined to obtain the second target request; The second target request is added to a preset queue by calling a preset function; Correspondingly, the preset error processing thread is called to process the second target request in the preset queue based on the second target request to locate the target storage and process the IO timeout fault of the located target storage. Processing the IO timeout fault of the located target storage, comprising:

5. The memory fault handling method of claim 4, wherein, A preset restart reset operation is performed to process the IO timeout fault of the located target storage; The preset restart reset operation comprises a restart operation and a reset operation of the target storage, a reset operation of a target channel bus and a reset operation of a target adapter; the target channel bus is a channel bus between the storage controller and the target storage, and the target adapter is an adapter between the storage controller and the target storage. After the IO timeout fault of the target storage is monitored, before the IO timeout fault of the target storage is processed, further comprising:

6. The memory fault handling method of claim 1, wherein, The state of the storage controller is changed from a first state to a second state; Correspondingly, after the IO timeout fault of the target storage is processed, further comprising: The state of the storage controller is changed from the second state to the first state; ​ The first state represents that the storage controller currently allows receiving new IO requests, and the second state represents that the storage controller currently refuses to receive any new IO request.

7. The memory failure handling method according to any one of claims 1 to 6, wherein After processing the IO timeout failure of the target storage, the method further includes: prohibiting the target storage from performing any IO operation, isolating the target storage, and marking the target storage as offline.

8. The memory fault handling method of claim 7, wherein, After marking the target storage as offline, the method further includes: if a preset condition is met, sending a preset control command to the target storage to control the target storage to attempt to perform an IO operation; monitoring a failure state of the target storage during the process in which the target storage attempts to perform the IO operation, and determining whether to isolate the target storage again based on the failure state; The preset condition includes that a continuous isolation duration of the target storage reaches a preset duration threshold and / or a maintenance record of the target storage is monitored.

9. The memory fault handling method of claim 8, wherein, Monitoring the failure state of the target storage and determining whether to isolate the target storage again based on the failure state includes: configuring a failure count value for the target storage, and an initial value of the failure count value is zero; if an IO timeout failure of the target storage is monitored, isolating the target storage again and updating the failure count value; if the failure count value is greater than a preset threshold, permanently isolating the target storage.

10. A memory failure handling apparatus, characterized by comprising: The storage controller includes: a function closing unit configured to control the target storage to close an IO retry function if an IO timeout failure of the target storage is monitored; an abort processing unit configured to start an abort operation on each first target request in the target storage and determine an abort operation result corresponding to each first target request, wherein the first target request is an unprocessed IO request in the target storage, and the abort operation result includes abort success or abort failure; an error processing unit configured to call a preset error processing thread and process the IO timeout failure of the target storage based on a second target request, wherein the second target request is the first target request with the abort operation result of abort failure.

11. A storage controller, comprising: The storage controller includes a processing unit and a storage unit, wherein the storage unit is configured to store a computer program, and the computer program is loaded and executed by the processing unit to implement the storage failure processing method according to any one of claims 1 to 9.

12. A storage system, characterized by The storage controller according to claim 11 and a plurality of storage devices connected to the storage controller.

13. A computer-readable storage medium, characterized in that, A computer program is stored, and the computer program is executed by a processor to implement the storage failure processing method according to any one of claims 1 to 9.

14. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by a processor to implement the storage failure processing method according to any one of claims 1 to 9.

Images