Balance resilience and performance by selectively using degraded writes and spare capacity in the storage system.

Abstract

A technique is provided for handling user input / output (I / O) write requests in a fault-tolerant data storage system (e.g., a RAID storage system) operating in degraded mode by selecting between performing a degraded write operation or a write operation to spare capacity. The method includes: receiving a user I / O write request, which includes data to be written to a RAID array operating in degraded mode; determining whether spare capacity has been allocated for rebuilding lost data on inaccessible storage devices of the RAID array; and determining whether lost data blocks associated with the I / O write request have been rebuilt to the spare capacity. When the lost data blocks associated with the data in the I / O write request have not yet been rebuilt to the allocated spare capacity, a degraded write operation is performed without using the spare capacity.

Description

Technical Field

[0001] This disclosure generally relates to data storage management in information processing systems, and more specifically, to data storage management techniques for enhancing the performance and reliability of data storage. Background Technology

[0002] Various technologies are used in data storage systems to provide resilience and fault correction to protect stored data in the event of storage device or storage node failure. Such technologies include erase coding and RAID (Redundant Array of Independent Drives) technology. Typically, RAID is a technology used to enhance the performance and reliability of data storage. Various RAID levels exist, implementing different combinations of technologies such as data mirroring, data striping, and parity checking to provide fault tolerance and allow recovery or otherwise reconstruction of lost data in the event of failure of storage devices such as hard disk drives (HDDs) and solid-state drives (SSDs).

[0003] For various applications and services, data storage systems typically require "five nines" availability (99.999% uptime) or "six nines" availability (99.9999% uptime). A key factor in providing such a high level of availability for RAID storage systems is the RAID control system's ability to correct errors much faster than they can occur; otherwise, too many device failures would lead to data loss and downtime. A certain amount of spare capacity is usually reserved, which the RAID control system can immediately utilize when storage devices fail. This reduces the time the RAID storage system is vulnerable to additional failures. Even when no spare capacity is available, a RAID storage system can operate in "degraded mode," as long as the number of existing failures does not exceed the maximum number of failures or fault tolerance level the RAID system is configured to support. When one or more devices fail in a RAID storage system, not all data (e.g., data stripes, parity data, etc.) can be written to the RAID array, as only existing data stripes or parity data on the non-failed storage devices can be written, resulting in what is known as "degraded writes." When a RAID storage system operates in degraded mode, there are performance impacts associated with using degraded writes to handle user input / output (I / O) write requests. Summary of the Invention

[0004] Exemplary embodiments of this disclosure include techniques for processing user I / O write requests in a fault-tolerant data storage system (e.g., a RAID storage system) by selecting between performing a degraded write operation or performing a write operation to spare capacity when the fault-tolerant data storage system is operating in a degraded mode. For example, one embodiment includes a method implemented by a storage control system. The storage control system receives a user I / O write request, which includes data to be written to a RAID array comprising multiple storage devices managed by the storage control system. The storage control system determines whether the RAID array is operating in a degraded mode, in which at least one of the multiple storage devices of the RAID array is inaccessible due to a failure. In response to determining that the RAID array is operating in a degraded mode, the storage control system determines (i) whether spare capacity has been allocated to the RAID array for rebuilding lost data from at least one inaccessible storage device of the RAID array, and (ii) whether lost data blocks associated with the data of the I / O write request have been rebuilt to the allocated spare capacity. In response to determining that spare capacity has been allocated to the RAID array but lost data blocks associated with the data in the I / O write request have not yet been reconstructed to the allocated spare capacity, the storage control system processes the user I / O write request by performing a degraded write operation using the remaining storage devices among the multiple storage devices in the RAID array, without using the spare capacity.

[0005] In another embodiment, in response to determining that the RAID array is operating in degraded mode, the storage control system determines whether the remaining resilience level of the RAID array is sufficient to provide fault tolerance for at least one additional failure of the remaining storage devices in the RAID array. Then, in response to determining that the remaining resilience level of the RAID array is sufficient to provide fault tolerance for at least one additional failure of the remaining storage devices in the RAID array, the storage control system uses the remaining storage devices from the plurality of storage devices in the RAID array to perform degraded write operations without using the allocated spare capacity. On the other hand, in response to determining that the remaining resilience level of the RAID array is insufficient to provide fault tolerance for at least one additional failure of the remaining storage devices in the RAID array, the storage control system processes user I / O write requests by performing write operations using the allocated spare capacity and the remaining storage devices in the RAID array.

[0006] Other embodiments of this disclosure include, but are not limited to, methods, systems, and articles of art that include processor-readable storage media, configured to process user I / O write requests in a fault-tolerant data storage system by selecting between performing a degraded write operation or performing a write operation to spare capacity when the data storage system is operating in a degraded mode. Attached Figure Description

[0007] Figure 1A and Figure 1B An exemplary embodiment of the present disclosure is illustrated, comprising an information processing system including a RAID storage system configured to process user I / O write requests when operating in a degraded mode by selecting between performing a degraded write operation or performing a write operation to spare capacity.

[0008] Figure 2 The illustration schematically shows an exemplary embodiment according to the present disclosure, in which... Figure 1A and Figure 1B The RAID 1 configuration implemented in the information processing system.

[0009] Figure 3 The illustration schematically shows an exemplary embodiment according to the present disclosure, in which... Figure 1A and Figure 1B The RAID 6 configuration implemented in the information processing system.

[0010] Figure 4A and Figure 4B A flowchart is shown of a method implemented in a RAID storage system according to an exemplary embodiment of the present disclosure, for processing user I / O write requests by selecting between performing a degrade write operation or performing a write operation to spare capacity.

[0011] Figure 5 The schematic illustration shows a framework for a server node for implementing a storage node hosting a RAID control module, according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0012] Exemplary embodiments of this disclosure will now be discussed in further detail with respect to systems and methods for implementing fault-tolerant data storage systems (e.g., RAID storage systems), configured to process user I / O write requests by selecting between performing a degraded write operation or performing a write operation to spare capacity when the fault-tolerant data storage system is operating in degraded mode. For illustrative purposes, exemplary embodiments will be described herein with reference to exemplary information processing systems implementing RAID data storage systems and associated computers, servers, storage devices, and other processing devices. However, it should be understood that these and other embodiments are not limited to the specific illustrative system and device configurations shown. Therefore, the term "information processing system" as used herein is intended to be interpreted broadly to cover, for example, processing systems including cloud computing and storage systems, as well as other types of processing systems including various combinations of physical and virtual processing resources. Thus, an information processing system may include, for example, at least one data center or other cloud-based system comprising one or more clouds hosting multiple tenants sharing cloud resources. Many different types of enterprise computing and storage systems are also covered by the term "information processing system" because the term is used extensively herein.

[0013] Figure 1A and Figure 1B An information processing system including a RAID storage system, according to an exemplary embodiment of the present disclosure, is schematically illustrated. The RAID storage system is configured to process user I / O write requests by selecting between performing a degraded write operation or performing a write operation to spare capacity when operating in degraded mode. More specifically, Figure 1A An information processing system 100 is schematically illustrated, comprising a plurality of computing nodes 110-1, 110-2, ..., 110-c (collectively referred to as computing nodes 110), one or more management nodes 115 (which support the management layer of system 100), a communication network 120, and a data storage system 130 (which supports the data storage layer of system 100). The data storage system 130 includes a plurality of storage nodes 140-1, 140-2, ..., 140-n (collectively referred to as storage nodes 140, or individually referred to as storage nodes 140). In the context of the exemplary embodiments discussed in further detail below, the management node 115 implements a data recovery control system 117, which is configured to perform various functions to support data recovery operations in the event of a failure of storage devices and / or storage nodes 140 of the data storage system 130. Figure 1B An exemplary framework of at least one or more of the storage nodes 140 is schematically shown.

[0014] Specifically, if Figure 1BAs shown, storage node 140 includes a storage control system 150, which includes a storage virtualization and management service module 160, a RAID control module 170, and a local storage device 180. The RAID control module 170 includes various modules, including but not limited to a data striping module 172, a parity calculation module 174, a rebuild control module 176, and a degradation write control module 178, the functions of which will be described in further detail below. In some embodiments, the storage control system 150 includes a software layer of a software-defined storage system, wherein the RAID control module 170 is an integrated software module of the storage control system 150 that implements a software RAID control system (e.g., a RAID drive). In other embodiments, the RAID control module 170 includes a separate software system that is separate from the storage control system 150 but configured to operate in conjunction with the storage control system 150. It should also be noted that the storage control system 150 may include additional modules and other components that are typically included in various implementations of a storage control system (e.g., a software-defined storage system), although such additional modules and other components are omitted for clarity and simplicity of illustration.

[0015] Compute node 110 schematically includes physical and / or virtual compute nodes that process data and perform workloads. For example, compute node 110 may include one or more server nodes (e.g., bare-metal server nodes) and / or one or more virtual machines. In some embodiments, compute node 110 includes a cluster of physical server nodes or other types of computers in an enterprise computer system, a cloud-based computing system, or other arrangements of multiple compute nodes associated with a respective user. In some embodiments, compute node 110 includes a cluster of virtual machines running on one or more physical server nodes.

[0016] Compute node 110 is configured to process data and perform tasks / workloads and computational work individually or in a distributed manner, thereby providing computing services, such as executing one or more applications on behalf of each of one or more users associated with a corresponding compute node in the compute node. Such an application schematically issues I / O requests processed by a corresponding storage node in storage node 140. As used herein, the term I / O request refers to at least one of input and output. For example, an I / O request may include write and / or read requests for stored data on a given storage node 140 of data storage system 130.

[0017] Compute node 110 is configured to write data to and read data from storage node 140 based on applications executed on the compute node by system users. Compute node 110 communicates with storage node 140 via communication network 120. Although communication network 120 is generally depicted in Figure 1, it should be understood that communication network 120 may include any known communication network, such as a global computer network (e.g., the Internet), a wide area network (WAN), a local area network (LAN), an intranet, a satellite network, a telephone or cable network, a cellular network, a wireless network such as Wi-Fi or WiMAX, a storage fabric (e.g., an Ethernet storage network), or various portions or combinations of these and other types of networks.

[0018] In this regard, the term "network" as used herein is therefore intended to be interpreted broadly to encompass a variety of different network arrangements, including combinations of multiple networks of different types, implemented using, for example, Transmission Control / Internet Protocol (TCP / IP) or other communication protocols such as Fibre Channel (FC), Ethernet FC (FCoE), Internet Small Computer System Interface (iSCSI), Peripheral Component Interconnect Fast (PCIe), InfiniBand, Gigabit Ethernet, etc., to implement I / O channels and support storage network connectivity. As those skilled in the art will understand, many alternative networking arrangements are possible in a given embodiment.

[0019] Data storage system 130 may include any type of data storage system or combination of data storage systems, including but not limited to Storage Area Network (SAN) systems, Network Attached Storage (NAS) systems, Direct Attached Storage (DAS) systems, and other types of data storage systems, including software-defined storage, clustered or distributed virtual and / or physical infrastructure. The term "data storage system" as used herein should be interpreted broadly and is not intended to be limited to any particular type of storage system. In some embodiments, the storage system may be implemented using a combination of storage nodes and compute nodes. In some embodiments, storage node 140 includes storage server nodes (e.g., Figure 5 The server node 500 shown herein has one or more processing devices, each having a processor and memory, and may implement virtual machines and / or containers, but multiple other configurations are possible. In some embodiments, one or more storage nodes in storage nodes 140 may additionally implement the functionality of a compute node, and vice versa. Therefore, the term "storage node" as used herein is intended to be interpreted broadly as a server node (e.g., a physical server machine) that includes local persistent storage devices (e.g., storage device 180) and storage control systems (e.g., control system 150, ...). Figure 1BIt is configured to manage and control access to local persistent storage devices. Storage nodes may further include one or more compute nodes to process data and perform tasks / workloads.

[0020] In some embodiments, such as Figure 1B As shown, storage node 140 is a physical server node or storage device, wherein storage device 180 includes DAS resources (internal and / or external storage resources), such as hard disk drives (HDDs), solid-state drives (SSDs), flash memory cards, or other types of non-volatile memory (NVM) devices, such as non-volatile random access memory (NVRAM), phase-change RAM (PC-RAM), and magnetic RAM (MRAM). These and various combinations of multiple different types of storage devices 180 can be implemented in storage node 140. In this regard, the term "storage device" as used herein is intended to be interpreted broadly to encompass, for example, SSDs, HDDs, flash drives, hybrid drives, or other types of storage media. Data storage device 180 is connected to storage node 140 using any suitable host interface (e.g., a host bus adapter) using a suitable protocol such as ATA, SATA, eSATA, NVMe, SCSI, SAS, etc. In other embodiments, storage node 140 may be a network connected to one or more NAS nodes via a local area network.

[0021] Storage control system 150 is configured to manage storage device 180 and control I / O access to storage device 180 and / or other storage resources (e.g., DAS or NAS resources) directly attached to or networked to storage node 140. In some embodiments, storage control system 150 is a component of a software-defined storage (SDS) system (e.g., a storage data server) that supports the virtualization of storage device 180 by decoupling control and management software from the underlying hardware architecture of storage device 180. Storage control system 150 runs on the server operating system of storage node 140 to provide storage services. More specifically, in a software-defined storage environment, storage control system 150 includes a storage data server configured to abstract storage access services from the underlying storage hardware, thereby controlling and managing I / O requests issued by compute node 110, and supporting networking and connectivity. In this example, the storage control system 150 includes a software layer hosted by the storage node 140 and deployed in the data path between the compute node 110 and the storage device 180 of the storage node 140. The storage control system 150 is configured to process I / O requests from the compute node 110 by accessing the storage device 180 to store / retrieve data from / from the storage device 180 based on the I / O requests.

[0022] In a software-defined storage environment, storage control system 150 includes a storage data server installed on each storage node that contributes its storage to the data storage system. Storage control system 150 is configured to provide, coordinate, and manage local storage resources (e.g., storage devices 180) of storage node 140. For example, storage control system 150 implements a method configured to create and manage storage pools (e.g., virtual pools of block storage) by aggregating capacity from storage devices 180. Storage control system 150 may partition the storage pool into one or more volumes and expose these volumes as virtual block devices to compute node 110. For example, a virtual block device may correspond to a volume of the storage pool. Each virtual block device includes any number of actual physical storage devices, wherein each block device is preferably homogeneous in terms of the type of storage devices constituting the block device (e.g., a block device may consist only of HDD devices or SSD devices, etc.).

[0023] In a software-defined storage environment Figure 1A Each storage node in the storage node 140 can run an instance of storage control system 150 to convert the corresponding local storage resources of storage node 140 (e.g., DAS storage devices and / or NAS storage devices) into local block storage. Each instance of storage control system 150 contributes some or all of its local block storage (HDD, SSD, PCIe, NVMe, and flash cards) to an aggregated storage pool of the storage server node cluster (e.g., the cluster 140 of storage nodes) to achieve a server-based SAN (e.g., a virtual SAN). In this configuration, each storage node 140 is part of a loosely coupled server cluster capable of “extending” the software-defined storage environment, where each instance of storage control system 150 running on a corresponding storage node in storage node 140 contributes its local storage space to an aggregated virtual pool of block storage with different performance tiers (e.g., HDD, SSD, etc.) within the virtual SAN.

[0024] In some embodiments, in addition to the storage control system 150 operating as a storage data server to create and expose volumes of the storage tier, the software-defined storage environment includes other components, such as (i) data storage clients that consume the storage tier and (ii) metadata managers that coordinate the storage tier. More specifically, on the client side (e.g., compute node 110), the SDS data storage client (SDC) is a lightweight block device driver (e.g., server node) deployed on each node that consumes the shared block storage volumes exposed by the storage control system 150 of storage node 140. Specifically, the SDC runs on the same host (e.g., server) as compute node 110 and needs to access the block devices exposed and managed by the storage control system 150 in storage node 140. The SDC exposes block devices representing virtual storage volumes currently mapped to that host. Specifically, the SDC acts as a block driver for the client (server), where the SDC intercepts I / O requests issued by compute node 110 and uses the intercepted I / O requests to access the block storage managed by the storage control system 150. SDC provides access to the operating system or hypervisor (which runs SDC) of logical block devices (e.g., volumes).

[0025] In a software-defined storage environment, the primary data and replicas of each storage volume are distributed across storage nodes 140, and each SDC knows which SDS data storage server (e.g., storage control system 150) holds its block data. Therefore, multipathing can be performed locally by the SDC. Specifically, each SDC knows how to direct I / O requests to the relevant destination SDS storage data server (e.g., storage control system 150). In this respect, there is no central point of routing, and each SDC performs its own routing independently of any other SDC. This implementation prevents unnecessary network traffic and redundant SDS resource usage. Each SDC maintains a peer-to-peer connection to each storage control system 150 managing the storage pool, so each SDC can communicate with all storage nodes 140 storing data via multiple paths. This multi-point peer-to-peer communication allows SDCs to simultaneously read and write data to and from all points, eliminating bottlenecks and enabling fast routing around faulty paths.

[0026] Figure 1AThe management node 115 implements a management layer configured to manage and configure the storage environment of system 100. In some embodiments, the management node 115 includes an SDS metadata manager component, wherein the management node 115 includes a tightly coupled cluster of nodes configured to supervise the operation of the storage cluster and manage the configuration of the storage cluster. The SDS metadata manager operates outside the data path and provides relevant information to SDS clients and storage servers to allow such a component to control data path operations. The SDS metadata manager is configured to manage the mapping from SDC data clients to SDS data storage servers. The SDS metadata manager manages various types of metadata required for the system operation of the SDS environment, such as configuration changes, management of SDS data clients and data servers, device mapping, values, snapshots, system capacity (including device allocation and / or capacity release), RAID protection, recovery from errors and failures, and system rebuild tasks, including rebalancing.

[0027] although Figure 1A An exemplary embodiment of a two-tier deployment is shown in which compute node 110 and storage node 140 are separate and connected via communication network 120. However, in other embodiments, a converged infrastructure (e.g., a hyperconverged infrastructure) can be implemented to integrate compute node 110, storage node 140, and communication network 120 together in an engineered system. For example, in a hyperconverged deployment, a single-tier deployment is implemented where storage data clients and storage data servers (e.g., storage control system 150) run on the same nodes (e.g., each node deploys both storage data clients and storage data servers), making each node both a data storage consumer and a data storage provider. In other embodiments, Figure 1A The system can be implemented using a combination of single-tier and two-tier deployments.

[0028] Regardless of the specific implementation of the storage environment, as mentioned above... Figure 1BThe various modules of the storage control system 150 collectively provide a data storage and management approach configured to perform various functions. Specifically, the storage virtualization and management service module 160 implements any suitable Logical Volume Management (LVM) system configured to create and manage local storage volumes by aggregating local storage devices 180 into one or more virtual storage pools. These virtual storage pools are thin-provisioned to maximize capacity, and each pool is logically divided into one or more storage volumes. These storage volumes are exposed to the compute node 110 as block devices (e.g., raw logical unit numbers (LUNs)) for storing data. In some embodiments, the storage device 180 is configured as a block storage device, where raw storage volumes are created, and each block can be controlled by the storage control system 150 as, for example, a separate disk drive. Depending on the needs of a given data storage system application, each block can be formatted individually using the same or different file systems. Storage pools are primarily used to group storage devices based on device type and performance. For example, SSDs are grouped into SSD pools, and HDDs are grouped into HDD pools. In addition, in some embodiments, the storage virtualization and management service module 160 implements methods to support various data storage management services, such as data protection, data migration, deduplication, replication, thin provisioning, snapshots, and data backup.

[0029] RAID control module 170 implements a control method configured to distribute, organize, and manage data across multiple storage devices in a RAID array according to a given RAID level configuration (e.g., RAID 1, RAID 6, etc.). RAID control module 170 of a given storage node 140 is configured to manage a RAID array comprising multiple storage devices such as HDDs or SSDs, including local storage device 180 of the given storage node 140 and... Figure 1A The data storage system 130 shown includes remote storage devices 180-R on other storage nodes 140. In some embodiments, the RAID control module 170 of a given storage node 140 is configured to communicate directly with remote storage devices 180-R on other storage nodes 140, which are portions of a RAID array managed and controlled by the RAID control module 170. In this respect, the RAID control module 170 can perform I / O read and I / O write operations on the remote storage devices 180-R, which are portions of a RAID array managed by the RAID control module 170.

[0030] Based on a given RAID level configuration, the RAID control module 170 is configured to use RAID storage technologies such as striping, mirroring, parity, and combinations thereof to control and manage the organization of data in a given RAID array. Specifically, the data striping module 172 is configured to support a RAID scheme that implements data striping technology to distribute data across multiple storage devices in the RAID array. Specifically, the data striping module 172 is configured to divide data into flexible units called "strips" and to divide the stripes into equally sized data "strips" stored on different storage devices within the RAID array.

[0031] The parity calculation module 174 is configured to support a parity-based RAID scheme. Specifically, the parity calculation module 174 is configured to use one or more types of parity functions (e.g., XOR functions, etc.) to calculate parity data (e.g., data bars). The type of parity function implemented will depend on the RAID level configuration supported by the RAID control module 170. The rebuild control module 176 is configured to control the rebuilding of spare capacity in the event of one or more failures of a given RAID array (e.g., failure of storage devices in the given RAID array, failure of a given storage node 140 including storage devices implemented in the given RAID array, etc.).

[0032] During periods when the RAID control module 170 operates a given RAID array in degraded mode due to one or more failures of storage devices and / or storage nodes associated with the given RAID array, the RAID control module 170 implements a method configured to provide a balance between I / O performance and data resilience when processing user I / O write requests. More specifically, when the RAID array is operating in degraded mode, the degraded write control module 178 is configured to make an intelligent decision for processing user I / O write requests by selecting between (i) performing a degraded write to the RAID array or (ii) performing a write operation to spare capacity. As explained in further detail below, such a determination is based on factors such as, for example, the amount of remaining resilience of the given RAID configuration when operating in degraded mode, whether spare capacity has been allocated and the current state of the rebuild process, and other relevant factors. When the RAID array is operating in degraded mode, the RAID control module 170 distinguishes between user I / O write requests to the RAID array processed by the rebuild control module 176 and I / O requests associated with the background rebuild process of the RAID array.

[0033] In some embodiments, the RAID control module 170 includes a software module of the storage control system 150, wherein the RAID function is performed using the operating system and existing hardware of the host storage node 140. For example, the software-based RAID control module 170 utilizes existing hardware such as a processor (e.g., CPU) and hardware controllers such as SAS, IDE, SATA, SCSI, and SSD controllers. Where a hardware RAID controller is implemented on the master storage node 140, the storage control system 150 can override the hardware RAID controller, allowing the software-based RAID control module 170 to control the RAID configuration and functions. In some embodiments, the software-based RAID control module 170 can utilize the read cache and write cache of the hardware RAID controller to support read and write operations on the RAID array.

[0034] As is known in the art, different RAID levels have different fault tolerance, data redundancy, and performance characteristics and attributes. Specifically, fault tolerance refers to the ability of a RAID array to withstand one or more failures without losing data. The performance of a given RAID level refers to the variation in read and write performance (e.g., speed) of a given RAID array compared to reading and writing to a single storage device. The capacity of a given RAID array refers to the amount of data that can be written to the RAID array. The capacity of a given RAID array is typically less than the total capacity of the storage devices within the given RAID array.

[0035] The exemplary embodiments of this disclosure can be implemented using various types of RAID level configurations. For illustrative purposes, the exemplary embodiments will be discussed in the context of RAID 1 and RAID 6 level implementations. For example, Figure 2 It schematically shows that it can be Figure 1A and Figure 1B The RAID 1 configuration 200 implemented in System 100. Generally, RAID 1 utilizes a data mirroring process to store data to a given storage device and stores mirrored copies of the data (alternatively called replicas) to one or more additional storage devices. Specifically, as... Figure 2 As shown, the RAID 1 configuration 200 includes three storage devices 210-1, 210-2, and 210-3, wherein each storage device 210-1, 210-2, and 210-3 stores a copy of each data block 1, block 2, block 3, etc. In some embodiments, such as Figure 2 As shown, each storage device 210-1, 210-2 and 210-3 is associated with a different storage node 220-1, 220-2 and 220-3, respectively.

[0036] RAID 1 configuration 200 uses data mirroring technology without parity to provide fault tolerance. More specifically, in Figure 2 In an exemplary embodiment, the RAID 1 configuration 200 provides resilience against the following failures: (i) a single failure, such as the failure of one of the storage devices 210-1, 210-2, and 210-3 or one of the storage nodes 220-1, 220-2, and 220-3; and (ii) two failures, such as the failure of two of the storage devices 210-1, 210-2, and 210-3 or two of the storage nodes 220-1, 220-2, and 220-3. In the event of a single or double failure, the RAID control module 170 can access copies of data blocks stored in the functional storage devices(s), such as block 1, block 2, block 3, etc. The capacity of the RAID 1 configuration is equal to the capacity of a single storage device within the RAID 1 array.

[0037] in this regard, Figure 2 The RAID 1 configuration 200 shown implements mirroring technology and three storage devices to provide fault tolerance for two failures. In practice, since each storage device 210-1, 210-2, and 210-3 has the same copy of data blocks, any I / O read or write request can be served by any one of the storage devices 210-1, 210-2, and 210-3 in the currently accessible and operational RAID array. In the event of a storage device or storage node failure, the RAID 1 configuration 200 can operate in degraded mode, provided that one storage device within the RAID array is accessible and operational. In other RAID embodiments such as RAID 5 and RAID 6, depending on the RAID configuration, distributed parity checking and striping of data across multiple storage devices are used to provide fault tolerance for one or more failures.

[0038] Specifically, RAID 5 and RAID 6 utilize a combination of block-level data striping and distributed parity techniques to provide resilience and error correction in the event of failures, such as storage device failures. The data striping process for RAID 5 and RAID 6 involves dividing the stored data into elastic units called "strips," and further dividing these data "strips" into multiple data blocks of equal size, also called "strips." The constituent stripes of a given stripe are stored in a given order on different storage devices within the RAID array. Additionally, a parity block is calculated for each data stripe and written to the data storage devices within the RAID array. The parity block for a data stripe is not written to a single storage device but is distributed across all storage devices within the RAID array. In the event that a constituent data stripe is lost due to storage device failure, the parity block for a given data stripe can be used to recalculate the data that made up the constituent data stripe.

[0039] RAID 5 uses a single parity function (e.g., the XOR function) to calculate the parity block for a given data stripe, while RAID 6 uses two independent parity functions to calculate two distinct parity blocks for a given data stripe. These two distinct parity blocks are written to two separate storage devices within the RAID 6 array. RAID 5 requires a minimum of three storage devices, while RAID 6 requires a minimum of four. The capacity of a RAID 5 array with N storage devices is equal to (N-1) times the size of the smallest storage device in the array (because the capacity of one storage device is used to store the parity function data for fault tolerance). The capacity of a RAID 6 array with N storage devices is equal to (N-2) times the size of the smallest storage device in the array (because the capacity of two storage devices is used to store the parity function data for fault tolerance).

[0040] For example, Figure 3 It schematically shows that it can be Figure 1A and Figure 1B The RAID 6 configuration implemented in System 100 is 300. Specifically, as... Figure 3 As shown, the RAID 6 configuration 300 includes four storage devices 310-1, 310-2, 310-3, and 310-4, each associated with a different storage node 320-1, 320-2, 320-3, and 320-4, respectively. Figure 3 In an exemplary embodiment, each data block (data stripe), namely block 1, block 2, block 3, block 4, etc., is divided into two constituent data blocks (data stripes) and has two independent parity check data blocks. Specifically, as... Figure 3 As shown, block 1 is divided into two constituent data blocks, block 1A and block 1B, and has two independent parity data blocks, parity check B1-1 and parity check B1-2 (e.g., P and Q parity check codes). Block 2 is divided into two constituent data blocks, block 2A and block 2B, and has two independent parity data blocks, parity check B2-1 and parity check B2-2. Block 3 is divided into two constituent data blocks, block 3A and block 3B, and has two independent parity data blocks, parity check B3-1 and parity check B3-2. Block 4 is divided into two constituent data blocks, block 4A and block 4B, and has two independent parity data blocks, parity check B4-1 and parity check B4-2.

[0041] Although Figure 3An exemplary RAID 6 layout for data blocks and parity blocks is shown, but different RAID 6 layout schemes can be implemented, specifying the order and location in which the constituent data blocks (strips) and associated parity blocks are written to storage devices 310-1, 310-2, 310-3, and 310-4 within the RAID 6 configuration 300. For each data stripe, the RAID configuration 300 includes at least four storage devices and two independent parity blocks to provide resilience and data protection in the event of two failures.

[0042] In some embodiments, when a failure occurs in a given storage device 180 of storage node 140 or when a failure occurs in a given storage node 140, system 100 ( Figure 1A The management node 115 will automatically start and perform the rebuild process. Storage device failure may occur in situations such as (i) the storage device suffers a hardware failure or malfunction, (ii) the associated hardware device driver is damaged or destroyed, (iii) there is a faulty connection (e.g., a bad cable or bus) connecting the storage device 180 to the storage node 140, (iv) there is a power failure of the storage device, or when (v) the storage node 140 controlling the storage device 180 crashes.

[0043] The data recovery control system 117 of management node 115 can initiate and manage different types of data recovery processes in the event of different types of failures. For example, when the failure of a given storage device 180 is a result of a hardware failure or malfunction of storage device 180, or a failure or malfunction of an associated storage node 140, the data recovery control system 117 can automatically initiate a "forward rebuild" process. The forward rebuild process involves, for example, allocating spare capacity for one or more storage devices within the cluster of storage node 140, and providing instructions to the storage control system 150 of storage node 140 to begin the rebuild process, to rebuild spare capacity for data with the failed storage device(s) using a copy or mirror copy of data on another storage device, or to rebuild lost data using existing data blocks and parity information, etc.

[0044] In some embodiments, when a failed storage device or storage node is associated with a given RAID array managed by a given RAID control module 170, the rebuild control module 176 will continue the background rebuild process according to the rebuild parameters / instructions provided by the management node 115 and using the spare capacity allocated by the management node 115. Depending on the given RAID level configuration, the RAID rebuild control module 176 will use the associated RAID algorithm and parity data, mirrored copies, etc., to rebuild the spare capacity with lost data.

[0045] In other embodiments, when a failure of a given storage device 180 or associated storage node 140 is the result of a repairable problem (e.g., a power failure), the data recovery control system 117 may automatically initiate a "backward rebuild" process. The backward rebuild process involves, for example, reattaching the storage device or storage node in the storage environment after the failure has been repaired, and then updating or otherwise synchronizing the data on the reattached storage device to include changes made to the data during the period when the storage device or associated storage node was inaccessible.

[0046] The data recovery control system 117 can determine whether to perform a forward or backward reconstruction operation based on the type of fault and the latency associated with certain reconstruction operations. For example, if the fault of a given storage device 180 or associated storage node 140 is repairable, the data recovery control system 117 can automatically initiate a backward reconstruction process when the problem is repaired, or initiate a forward reconstruction operation to rebuild the lost data on spare capacity if performing a forward reconstruction operation would result in a faster recovery time than performing a backward reconstruction operation.

[0047] In some embodiments, management node 115 ( Figure 1A The management node 115 will monitor the "heartbeat" from storage node 140, and when no heartbeat is detected from a given storage node 140, the management node 115 will determine that the given storage node 140 has failed. The data recovery control system 117 will initiate a reconstruction process to protect the data, and the management node 115 may perform additional operations as needed to reconfigure the storage environment in response to the detected storage node failure. In other embodiments, when the storage control system 150 or RAID control module 170 of a given storage node 140 detects an I / O failure while reading from or writing to a given storage device 180 of the given storage node 140, the storage control system 150 will provide the management node 115 with a notification of a storage device I / O error returned for the given storage device. In response to this notification, the data recovery control system 117 will initiate a reconstruction process to protect the data, and the management node 115 may perform additional operations as needed to reconfigure the storage environment in response to the detected storage device failure.

[0048] When using spare capacity as part of a background rebuild operation, the performance impact must be considered. For example, when spare capacity is allocated, it initially contains no data. When new data is written to the spare capacity, the write operation needs to update the metadata associated with the newly written data, indicating which parts of the spare capacity are valid and which are invalid. The size of the metadata can impact performance because the granularity of metadata is typically larger than that of I / O. For example, a storage system might be configured such that metadata can only track changes at a 32KB granularity, while the I / O granularity is 4KB. In this case, 28KB needs to be read and combined with the user's 4KB, creating a larger 32KB write that can be referenced by the metadata, thus introducing performance overhead. The term "working area" as used in this article refers to an area at the metadata granularity (e.g., 32KB).

[0049] Furthermore, especially when spare capacity is shared across multiple storage nodes, its allocation is not an immediate process. In this case, actual capacity is allocated only in the event of a failure, as any storage node requiring spare capacity can use it when a failure occurs. During a failure, or even multiple failures, there will be numerous requests from the spare allocation mechanism, which can lead to bottlenecks and slow down user I / O. For RAID background rebuild operations, the time taken to complete the rebuild process will vary depending on factors such as the number of failed storage devices, the capacity of the failed storage devices that must be rebuilt, and the bandwidth allocated to perform the I / O rebuild operation compared to performing user I / O operations.

[0050] Specifically, the failure of a high-capacity storage device (e.g., terabyte-level capacity) or multiple high-capacity storage devices can result in a relatively lengthy rebuild operation that may take hours or days to complete. Furthermore, in some embodiments, when the RAID rebuild process is initiated, the data recovery control system 117 can specify a rebuild throttling strategy associated with the RAID background rebuild process. The rebuild throttling strategy for the RAID rebuild process determines the priority of performing rebuild I / O operations versus performing user / application I / O operations when accessing storage devices. Different rebuild throttling strategies can be applied, which (i) do not restrict rebuild I / O operations, (ii) limit concurrent I / O operations on a given storage node, (iii) favor user / application I / O operations, and (iv) implement dynamic bandwidth throttling, etc. Applying rebuild throttling will increase the time required to complete the rebuild operation and keep the RAID system in a degraded operating mode while reducing the impact on user I / O performance.

[0051] Furthermore, the performance impact of performing degraded write operations must be considered, as it varies depending on the given RAID configuration. For example, for RAID 1, there is no performance degradation when performing a degraded write operation. Specifically, a degraded write operation for a RAID 1 configuration with N storage devices involves writing data to existing storage devices (as associated with I / O write requests) when one or more of the N storage devices fail or are otherwise unavailable. In this case, the degraded write operation involves writing fewer than N copies of the data to the storage devices.

[0052] For example, in Figure 2 In the exemplary RAID 1 configuration 200, it is assumed that storage device 210-3 fails or becomes unavailable due to the failure of storage node 220-3, thus placing the RAID 1 configuration into a degraded mode. In this failure state, degraded write operations would involve (i) writing data to storage device 210-1 of the primary storage node 220-1, and (ii) writing a mirror copy to storage device 210-2 of the mirror storage node 220-2. In this case, there is no penalty associated with degraded write operations when data is written only to storage devices 210-1 and 210-2 and not to the failed / unavailable storage device 210-3. Furthermore, there is no performance penalty associated with subsequent read operations performed in response to I / O read requests for previously written data, because the requested data (e.g., block 1) can be read from one of storage devices 210-1 and 210-2 without needing to rebuild such requested data as would be in a RAID configuration utilizing parity data for fault tolerance.

[0053] On the other hand, for parity-based RAID configurations, there is a performance penalty associated with degraded write operations. Specifically, when a user issues an I / O read request to read data from a failed storage device, there is a performance penalty, which requires the RAID controller to use the remaining data (e.g., parity data) on the currently available storage devices within the RAID array to reconstruct the requested data.

[0054] For example, in Figure 3In the exemplary RAID 6 configuration 300, it is assumed that storage device 310-1 fails or becomes unavailable due to the failure of storage node 320-1, thereby placing the RAID 6 configuration 300 into a degraded mode. Further, it is assumed that an I / O write request to write data to data block 1A has been received. In this failure state, since data block 1A is unavailable, the degraded write operation will involve (i) reading data block 1B and parity data blocks, parity B1-1 and parity B2-1, (ii) using the received I / O write data and the read data block 1B to calculate new parity data for parity blocks, parity B1-1 and parity B2-1, and (iii) writing the new parity data blocks, parity B1-1 and parity B2-1, to storage devices 310-3 and 310-4.

[0055] Using a degraded write operation, the updated data block 1A is not written to the storage device. Instead, new parity data blocks, parity B1-1 and parity B2-1, are written, including the information needed to reconstruct the updated data block 1A to serve subsequent I / O read requests for the updated data block 1A. Specifically, in response to an I / O read request for the updated data block 1A, a degraded read operation is performed, which involves (i) reading data block 1B and the new parity data blocks, parity B1-1 and parity B2-1, (ii) using data block 1B and the new parity data blocks, parity B1-1 and parity B2-1 to reconstruct data block 1A, and (iii) returning the reconstructed data block 1A to serve the I / O read request. In this respect, there is a performance penalty associated with performing a degraded read operation to read data from a failed storage device in the RAID array, because the remaining data on the currently available storage devices in the RAID array must be used to reconstruct the data.

[0056] As described above, when processing user I / O write requests during the period when the RAID control module 170 operates the RAID array in degraded mode, the RAID control module 170 implements a method configured to provide a balance between I / O performance and data resilience. When the RAID array operates in degraded mode, the degraded write control module 178 is configured to make an intelligent decision for processing user I / O write requests by selecting between (i) performing a degraded write to the RAID array or (ii) performing a write operation to the spare capacity. In this configuration, the RAID control module 170 distinguishes between user I / O write requests to the RAID array and I / O requests associated with the background rebuild process of the RAID array processed by the rebuild control module 176.

[0057] In some embodiments, the degradation write control module 178 makes a determination on how to handle user I / O write requests based in part on the amount of remaining resilience for a given RAID configuration when operating in degradation mode. For example, for a RAID array providing resilience against two failures, the decision to perform a degradation I / O write operation instead of a write operation to spare capacity will depend on whether the RAID array is in a degradation state for one failure (in which case there is a fault tolerance level for the other failure) or in a degradation state for two failures (in which case the RAID array cannot protect against the other failure). When the RAID array is in a degradation mode with some remaining resilience, the RAID control module 170 can be configured such that a background rebuild process performed solely by the rebuild control module 176 allocates and writes spare capacity to obtain resilience. The background process is insensitive to latency and can tolerate performance overhead.

[0058] On the other hand, for performance-sensitive user I / O write requests, the RAID control module 170 only processes the user I / O request by writing to the spare capacity if the background process of the allocated spare capacity and the data block associated with the I / O write request have been rebuilt on the spare capacity. Otherwise, the RAID control module 170 performs a degraded write operation by writing the data associated with the user I / O request to a functionally accessible storage device of the RAID array. In the event of two failures (without residual protection against the other failure), resilience becomes critical, and all user I / O write requests will be processed by writing data to the RAID array using the spare capacity.

[0059] Figure 4A and 4B A flowchart of a method implemented in a RAID storage system according to an exemplary embodiment of the present disclosure is shown, the method being used to process user I / O write requests by selecting between performing a degraded write operation or performing a write operation to spare capacity when operating in degrade mode. In some embodiments, Figure 4A and Figure 4B It shows Figure 1B The exemplary operating mode of the RAID control module 170 is provided to offer a balance between I / O performance and data resilience when processing user I / O write requests during periods when the RAID control module 170 operates the RAID array in degraded mode. For illustrative purposes, [the following will be shown]. Figure 1A and Figure 1B The process flow is discussed in the context of the information processing system 100.

[0060] refer to Figure 4AThe RAID control module 170 receives user I / O write requests (block 400) to write data to the RAID array. As described above, in some embodiments, I / O requests for a given compute node 110 are serviced by an SDC running on the same server as the compute node, and when the I / O request is a write request, the SDC sends the I / O write request to the storage node 140 where the primary copy of the data resides, or to the storage node 140 having a RAID control module 170 assigned to handle data striping and parity calculations for the given RAID array.

[0061] Determine whether the RAID array is operating in degraded mode (box 401). If it is determined that the RAID array is not operating in degraded mode (negative determination in box 401), the RAID control module 170 will perform normal write operations according to the given RAID array configuration (box 402). For example, for a RAID 1 configuration, the RAID control module 170 will continue to write the data of the I / O write request to the local storage device (e.g., the primary replica) and write a mirror copy of the data to a remote storage device within the RAID array. For a RAID 6 configuration, the RAID control module 170 will perform the necessary data striping and parity calculation functions to write data to the local and remote storage devices of the RAID array.

[0062] On the other hand, if it is determined that the RAID array is operating in degraded mode (a positive determination in box 401), the RAID control module 170 will continue to determine whether to perform a degrade write operation or to perform a write operation using spare capacity. Such determination is based on factors such as the amount of remaining flexibility for a given RAID configuration when operating in degraded mode (decision box 403), whether spare capacity has been allocated and the current state of the rebuild process (decision box 404), and other relevant factors.

[0063] When at least one failure associated with a storage device or storage node of the RAID array occurs, the RAID array will operate in degraded mode. User I / O write requests may be received even when the RAID array is already operating in degraded mode due to one or more failures of the storage device and / or storage node associated with the RAID array (in box 400). In this case, the RAID control module 170 may be in the process of performing a rebuild operation instructed by the management node 115 using the spare capacity allocated by the management node 115 for the rebuild operation.

[0064] In other instances, I / O write requests may be received (in block 400) when the RAID array is not operating in degraded mode, but a normal write operation (block 402) fails and the RAID control module 170 receives an I / O error (e.g., a storage device I / O error) as a result of the write operation failure. In some embodiments, when an I / O error is returned in response to an attempted write operation to the RAID array, the RAID control module 170 will continue to execute any suitable procedures for handling I / O errors implemented by the storage control system 150 and / or the RAID control module 170.

[0065] Furthermore, the RAID control module 170 can further determine whether the I / O error is the result of a storage device failure, storage node failure, network failure, etc. As mentioned above, there are many possible causes for I / O errors, such as storage device hardware failure, faulty connections, power failure of the storage device, failure of the remote storage node 140, etc. The RAID control module 170 can make such a determination based on various factors, including but not limited to, receiving numerous I / O errors for write operations on a given storage device (within a relatively short period of time), receiving notification from the management node 115 regarding a given failure of the storage device, storage node, or communication network that caused such I / O errors, or other types of conditions or indications that enable the RAID control module 170 to determine that the storage devices of the RAID array have failed or are otherwise unavailable.

[0066] In some embodiments, as described above, when an I / O error occurs, the storage control system 150 provides a notification of the I / O error to the management layer (e.g., management node 115). In response to this notification, the management layer will automatically perform certain operations as needed to reconfigure the storage environment and recover from the I / O error. Such operations include, for example, providing instructions to one or more storage control systems to continue rebuilding the failed storage device or failed storage node (e.g., a failed server), providing instructions to the RAID control module to rebuild the data in the RAID array using spare capacity allocated by the management layer, reconfiguring the mapping metadata of the storage environment so that clients direct I / O requests to another storage control system (of another storage node 140), which includes storage devices with backups or copies of the data stored in the failed storage device, etc.

[0067] When the RAID control module 170 determines that an I / O error is the result of some fault associated with the RAID array, the RAID control module 170 may determine that the RAID array is in a degraded operation state and continue to process user I / O requests for the RAID array in a degraded mode. In the degraded mode, depending on the number of concurrently present faults at a given time and the fault tolerance of a given RAID array configuration (e.g., the maximum number of concurrently present faults allowed), the RAID array remains operational but with reduced fault tolerance or no longer has fault tolerance.

[0068] In all cases where the RAID array is considered to be operating in the degraded mode (positive determination in box 401), the RAID control module 170 will continue to determine whether to (i) perform a degraded write operation or (ii) use spare capacity to perform the write operation. In some embodiments, such determination is based in part on determining whether the remaining resilience of the RAID array is sufficient to provide the desired level of fault tolerance (determination in box 403). For example, in some embodiments, such determination is made based on comparing the number of concurrently present faults (f) in the RAID array with the maximum number of concurrently present faults (F) that the RAID array can handle without losing data. In some embodiments, when the number of concurrently present faults is less than the maximum number of concurrently present faults that the RAID array can handle without losing data (i.e., when f < F), the remaining resilience of the RAID array will be considered sufficient (positive determination in box 403).

[0069] For example, Figure 2 the exemplary RAID 1 configuration 200 shown provides fault tolerance for up to two concurrently present faults (F = 2). Similarly, Figure 3 the exemplary RAID 6 configuration 300 shown provides fault tolerance for up to two concurrently present faults (F = 2). In such embodiments, when there is one present fault (i.e., f = 1), the remaining resilience of the RAID array may be considered sufficient (positive determination in box 403), while when there are two concurrently present faults (i.e., f = 2), the remaining resilience of the RAID array will be considered insufficient (negative determination in box 403).

[0070] In other embodiments, the RAID array can be configured to provide fault tolerance for more than two concurrent failures (e.g., F = 3, 4, 5, …) in certain applications where such a high level of fault tolerance is required. In such embodiments, when the number (f) of concurrently existing failures in the RAID array equals the maximum number (F) of concurrently existing failures that the RAID array can handle without losing data (i.e., when f = F, where F = 3, 4, 5, …), the remaining resilience of the RAID array can be considered insufficient. In other embodiments, when the RAID array is configured to provide fault tolerance for more than two concurrent failures (e.g., F = 3, 4, 5, etc.), the remaining resilience of the RAID array can be considered insufficient depending on the given application when the number (f) of concurrently existing failures in the RAID array is less than the maximum number (F) of concurrent failures that the RAID array can handle, e.g., when f = F - 1, f = F - 2, etc.

[0071] Specifically, a determination (at block 403) can be made as to whether the remaining resilience of a given RAID array is sufficient or insufficient based on a resilience threshold (f Threshold ), which is set (statically or dynamically) based on various operating characteristics of the RAID array. For example, in a fault-tolerant storage system where the storage device failure rate is relatively high and the time to perform a background rebuild process is relatively long, it may be desirable to set the resilience threshold to some value less than F (i.e., f Threshold < F) to ensure that the RAID control system can perform a degraded write while still preventing data loss in the case where, at some point in time, the number (f) of concurrent failures exceeds the maximum number (F) of concurrent failures that the RAID array can handle before the background rebuild process can be completed for one or more failed storage devices.

[0072] For example, assume a given RAID array has fault tolerance for up to four failures (i.e., F = 4) and the resilience threshold is set to three (i.e., f Threshold = 3). In this example, a determination (at block 403) can be made as to whether the remaining resilience of the given RAID array is sufficient or insufficient based on comparing the number (f) of concurrently existing failures with the resilience threshold (f Threshold ). When the number of concurrently existing failures is less than the resilience threshold (i.e., f < f Threshold ), the remaining resilience of the RAID array can be considered sufficient (a positive determination at block 403). On the other hand, when the number of concurrently existing failures is equal to or greater than the resilience threshold (i.e., f ≥ f Threshold ), the remaining resilience of the RAID array can be considered insufficient (a negative determination at block 403). In an exemplary embodiment where F = 4, when the number of concurrently existing failures is determined to be equal to the resilience threshold (e.g., f = f Threshold=3), one (1) fault (i.e., Ff) Threshold The remaining elasticity of 4-3=1) can be considered insufficient for the purpose of performing a degraded write operation, while preventing potential data loss due to one or more additional failures before the background reconstruction process is completed.

[0073] In some embodiments, the resilience threshold (f) of a given fault-tolerant storage system can be determined based on statistical data and calculations associated with hardware and software failures. Threshold For example, the hardware failure characteristics of a storage device can be determined as a function of the failure rate over time. Furthermore, the availability of hardware and software components can be determined based on reliability parameters or metrics, including but not limited to, mean time between failures (MTBF), time to failure (FIT) (which represents the failure rate of 1 in 1 billion hours), mean time to repair (MTTR), etc. In this regard, it should be understood that various statistical methods and reliability parameters known in the art can be used to calculate the resilience threshold (f) of a given fault-tolerant storage system. Threshold Furthermore, this threshold will vary based on the availability and reliability characteristics determined by statistics associated with the various hardware and software components used in the fault-tolerant data storage system.

[0074] Return to reference Figure 4A When the remaining resilience of the RAID array is deemed sufficient (a positive determination in block 403), it is determined whether spare capacity has been allocated and whether the lost data of the RAID array has been used for reconstruction (a determination in block 404). As described above, in some embodiments, when a given storage device or storage node fails, the management layer continues to allocate spare capacity to the existing storage devices of the given storage node and initiates the reconstruction process by instructing the storage control system to reconstruct the lost data of the failed storage device(s) and / or storage node(s) associated with the given RAID array and to repopulate the allocated spare capacity with the reconstructed data. For example, for Figure 2 The exemplary RAID 1 configuration 200 shown illustrates a rebuild operation involving storing a mirror copy of the data on the spare capacity using existing data from a given storage device in the RAID array. For Figure 3 An exemplary RAID 6 configuration 300, the rebuild operation involves, for example, using P and Q parity checks to rebuild data on the failed storage device as well as data on the remaining storage devices of the RAID array.

[0075] When spare capacity has been allocated and rebuilt using the lost data from the RAID array (a positive determination in box 404), the spare capacity can be used to perform write operations (box 405). In some embodiments, upon receiving a user I / O write request (in box 400), the reconstruction process may be in an intermediate stage where a portion of the lost data has been copied or otherwise reconstructed and repopulated on the spare capacity. In this case, the “working area” of the given data block associated with the received user I / O write request may have already been rebuilt on the spare capacity, in which case the spare capacity can be used to perform write operations (box 405). On the other hand, the “working area” of the given data block associated with the received user I / O write request may not have yet been rebuilt on the spare capacity. In this case, although the spare capacity can be allocated and available, the spare capacity may only have been partially rebuilt and does not yet have the rebuilt data and associated metadata associated with the received user I / O write request (leading to a negative determination in box 404).

[0076] When it is determined that spare capacity has been allocated but the data associated with the I / O write request has not yet been reconstructed on the spare capacity (negative determination in box 404), the RAID array control module 170 may request access to the spare capacity and wait to perform the write operation using the spare capacity when a service request is made (box 406). If the waiting delay exceeds a predetermined delay threshold (positive determination in box 407), the RAID control module 170 will continue to perform a degraded write operation (box 408). On the other hand, if it is determined that the I / O request is or can be serviced by a delay not exceeding the delay threshold (negative determination in box 407), the spare capacity is used to perform the write operation.

[0077] In some embodiments, the process flows of blocks 406 and 407 are not implemented. For example, if spare capacity has not yet been allocated or is otherwise unavailable, the RAID control module 170 will continue to perform the degradation write operation (block 408), as... Figure 4A The dashed lines from boxes 404 to 408 are shown in the diagram. In other embodiments, when the RAID array includes a RAID 1 mirrored configuration, if spare capacity has been allocated, but the data associated with the I / O write request has not yet been reconstructed on the spare capacity (resulting in a negative determination in box 404), the RAID array control module 170 will continue to perform the degraded write operation (box 408). This alternative process flow is... Figure 4A The dashed lines from boxes 404 to 408 represent this alternative process. This process is based on setting the latency threshold of the RAID 1 array configuration to zero, because for a RAID 1 mirrored configuration, there is no latency associated with performing a degradation write operation.

[0078] In practice, for RAID 1 configurations, performing write operations using spare capacity (box 405) by waiting for a rebuild operation (in box 406) does not provide an I / O performance advantage compared to performing a degraded write operation (box 408). Specifically, for RAID 1 configurations, a performance advantage cannot be achieved by avoiding degraded write operations for a given data block because, unlike parity-based RAID configurations, subsequent read operations for a given data block do not require parity checks and / or data computations to read the given data block that was previously written to the RAID 1 array using a degraded write. In fact, subsequent operations reading data blocks from the RAID 1 array simply require reading data directly from the primary or mirrored storage device.

[0079] In this scenario, for a RAID 1 array configuration, instead of waiting to utilize spare capacity to fulfill user I / O write requests, the RAID control module 170 will continue to perform degraded write operations (box 408) to serve user I / O write requests with minimal latency. Ultimately, the I / O write data associated with the degraded write operation will be written to the spare capacity using the RAID rebuild I / O write operation during the background rebuild process.

[0080] On the other hand, in embodiments implementing parity-based RAID array configurations (e.g., RAID 6 configuration 300), Figure 3 The process flows in boxes 406 and 407 can be executed to improve the performance of executing user I / O write requests. For example, there may be a situation where the latency associated with performing a write operation using spare capacity is greater than the latency associated with performing a degraded write operation. In this case, when waiting for spare capacity and the latency of performing a write operation using spare capacity is greater than the latency of performing a degraded write operation (as confirmed in box 407), a degraded write operation (box 408) is performed to serve the user I / O write request. In this case, the I / O write data associated with the degraded write operation will be written to the spare capacity using the RAID reconstruction I / O write operation during the background reconstruction process.

[0081] On the other hand, there may be cases where the latency associated with performing a write operation using the spare capacity is less than or substantially equal to the latency associated with performing a degraded write operation. In such cases, when it is determined that waiting for the spare capacity and the latency of performing a write operation using the spare capacity does not exceed the latency of performing a degraded write operation (negative determination in box 407), the user I / O write request will be served by performing the write operation using the spare capacity (box 405).

[0082] In some embodiments, the latency threshold implemented for the determination process of decision block 407 may be a predetermined timeout value or a dynamically allocated timeout value, which is adjusted to take into account the overall system performance of the background RAID array rebuild process and different throttling rates. Furthermore, in some embodiments, depending on the application, the latency threshold may be determined at least in part based on the performance advantage achieved for subsequent read operations in a parity-based RAID configuration, which is obtained by waiting for the rebuild operation (in block 406) and performing the write operation (in block 405) using a RAID array with spare capacity.

[0083] In practice, when performing an I / O write operation on a given data block using spare capacity (box 405), a performance advantage is realized for subsequent I / O read requests for that data block because the data block can be read from the RAID array (e.g., using spare capacity) without having to perform the data / parity calculations required as part of the I / O read operation to reconstruct the lost data block. Conversely, if data is written to the RAID array using a degraded write operation (box 408), subsequent I / O read operations that read such data may require (i) reading the associated data strips and / or parity blocks from the existing storage devices in the RAID array, and (ii) performing the calculations required to reconstruct the lost data using the data strips and parity data. Therefore, writing data to the RAID array using a degraded write operation results in reduced I / O performance for subsequent read operations on the data due to the increased latency associated with having to read the existing data strips and parity and perform the calculations required to reconstruct the lost data as needed.

[0084] Returning to decision box 403, if it is determined that the remaining resilience of the RAID array is insufficient to provide the desired level of fault tolerance (negative determination in box 403), the process flow proceeds to... Figure 4B Box 409, where it is determined whether spare capacity has been allocated and whether the spare capacity is rebuilt using the lost data of the RAID array (decision box 409). In some embodiments, as described above, when the number of concurrent failures (f) of the RAID array is equal to the maximum number of concurrent failures (F) that the RAID array can handle without data loss (i.e., when f = F), the RAID array is considered to have insufficient remaining elasticity. In other embodiments, when the number of concurrent failures (f) of the RAID array is less than the maximum number of concurrent failures (F) that the RAID array can handle without data loss, for example when f = F-1, f = F-2, the RAID array may be considered to have insufficient remaining elasticity, depending on, for example, an elasticity threshold (f) specified for a given storage system. Threshold ).

[0085] If it is determined that spare capacity has been allocated and the spare capacity has been rebuilt using the lost data of the RAID array (a positive determination in box 409), pending user I / O write requests can be served by performing write operations on the RAID array using the spare capacity (box 410). When it is determined that spare capacity has been allocated but the data associated with the I / O write request has not yet been rebuilt on the spare capacity (a negative determination in box 409), the RAID array control module 170 will request access to the spare capacity and wait to perform write operations using the spare capacity when the request is served (box 411). In this case, assuming that the remaining elasticity is considered insufficient, the RAID control module 170 will continue to maximize fault tolerance and redundancy at the expense of I / O write performance, regardless of latency, by waiting to perform write operations on the RAID array using the spare capacity. In some embodiments, if it is determined (in box 409) that the spare capacity has not been allocated or is otherwise unavailable, the RAID control module 170 will continue to perform degraded write operations (box 413), such as... Figure 4A The dashed line from box 404 to box 408 is shown in the middle.

[0086] When the RAID control module 170 awaits access to spare capacity (in box 411) to perform a write operation for a pending user I / O write request, if it is determined that the spare capacity has been exhausted (affirmative determination in box 412), the RAID control module 170 will continue to perform a degraded write operation (box 413). On the other hand, if the RAID control module 170 is granted access to perform a write operation using the spare capacity, and the spare capacity has not been exhausted (negative determination in box 412), the RAID control module 170 will continue to serve pending user I / O write requests by performing write operations on the RAID array using the spare capacity (box 410).

[0087] It should be understood that Figure 4A and Figure 4B The process flow involves the processing performed to handle user I / O write requests. During the background RAID rebuild process, as part of the rebuild process, the RAID control module performs RAID rebuild I / O read and write operations to, for example, reconstruct lost data from one or more failed storage devices or storage nodes associated with the RAID array on the spare capacity. In this regard, as described above, during the execution of the background RAID rebuild process, the management layer can allocate I / O bandwidth between I / O requests to the spare capacity and rebuild the I / O requests to the spare capacity.

[0088] Furthermore, as mentioned above, when a RAID storage system operates in degraded mode, various factors (e.g., the number of existing faults, latency, etc.) are considered when determining whether to handle user I / O write requests in the RAID storage system by performing degraded write operations or by writing to spare capacity. This determination aims to strike a balance between fault tolerance and I / O performance in a RAID data storage system operating in degraded mode when: degraded write operations can be tolerated when there are a relatively small number of existing faults until a background rebuild operation refills the spare capacity with lost data; and in other cases where there are a relatively large number of existing faults with minimal or no fault tolerance, where the system should maximize redundancy at the expense of I / O performance. While conventional RAID systems typically use spare capacity to handle user I / O write requests when it is available, embodiments of this disclosure discussed herein will selectively utilize degraded write operations to handle user I / O write requests, even when spare capacity remains available, provided that degraded write operations offer better I / O performance compared to waiting to write to spare capacity.

[0089] Figure 5 The illustration schematically depicts an exemplary embodiment of the present disclosure for implementing a storage node (e.g., storage node 140). Figure 1A and 1B The server node 500 comprises a processor 502, storage interface circuitry 504, network interface circuitry 506, virtualization resources 508, system memory 510, and storage resources 516. System memory 510 includes volatile memory 512 and non-volatile memory 514. Processor 502 includes one or more types of hardware processors configured to process program instructions and data to execute a native operating system (OS) and applications running on the server node 500.

[0090] For example, processor 502 may include one or more CPUs, microprocessors, microcontrollers, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and other types of processors, as well as portions or combinations of such processors. As used herein, the term "processor" is broadly interpreted to include any type of processor that performs processing functions based on software, hardware, firmware, etc. For example, "processor" is broadly interpreted to include all types of hardware processors, including, for example, (i) general-purpose processors that include "performance cores" (e.g., low-latency cores), and (ii) workload-optimized processors that include any possible combination of multiple "throughput cores" and / or multiple hardware-based accelerators. Examples of workload-optimized processors include, for example, graphics processing units (GPUs), digital signal processors (DSPs), system-on-a-chip (SoCs), tensor processing units (TPUs), image processing units (IPUs), deep learning accelerators (DLAs), artificial intelligence (AI) accelerators, and other types of dedicated processors or coprocessors configured to perform one or more fixed functions.

[0091] Storage interface circuitry 504 enables processor 502 to read or write data to volatile and non-volatile memory / storage devices using one or more standard communication and / or storage control protocols, thereby interfacing and communicating with system memory 510, storage resources 516, and other local storage and non-infrastructure storage media. Such protocols include, but are not limited to, Non-Volatile Memory Fast (NVMe), Peripheral Component Interconnect Fast (PCIe), Parallel ATA (PATA), Serial ATA (SATA), Serial Attached SCSI (SAS), Fibre Channel, etc. Network interface circuitry 506 enables server node 500 to interface and communicate with the network and other system components. Network interface circuitry 506 includes network controllers such as network interface cards (NICs) and resources (e.g., network interface controllers (NICs) (e.g., SmartNICs, RDMA-enabled NICs), host bus adapter (HBA) cards, host channel adapter (HCA) cards, I / O adapters, converged Ethernet adapters, etc.) to support communication protocols and interfaces, including but not limited to PCIe, DMA, and RDMA data transfer protocols.

[0092] Virtualization resource 508 can be instantiated to perform one or more services or functions hosted by server node 500. For example, virtualization resource 508 can be configured to implement the storage control system 150 discussed herein. Figure 1BThe virtualization resource 508 includes various modules and functions. In one embodiment, the virtualization resource 508 includes virtual machines implemented using a hypervisor platform executed on server node 500, wherein one or more virtual machines can be instantiated to perform the functions of server node 500. As is known in the art, a virtual machine is a logical processing element that can be instantiated on one or more physical processing elements (e.g., servers, computers, or other processing devices). That is, a "virtual machine" generally refers to a software implementation of a machine (i.e., a computer) that executes programs in a manner similar to a physical machine. Therefore, different virtual machines can run different operating systems and multiple applications on the same physical computer.

[0093] A hypervisor is a more general example of what is referred to as "virtualized infrastructure." The hypervisor runs on the physical infrastructure of server nodes 500, such as CPUs and / or storage devices, and emulates the host system's CPU, memory, hard disks, network, and other hardware resources, enabling multiple virtual machines to share resources. The hypervisor can emulate multiple virtual hardware platforms isolated from each other, allowing virtual machines to run operating systems such as Linux and Windows servers on the same underlying physical host. The underlying physical infrastructure can include one or more commercially available distributed processing platforms suitable for the target application.

[0094] In another embodiment, virtualization resource 508 includes containers such as Docker containers or other types of Linux containers (LXC). As is known in the art, in a container-based application framework, each application container includes a separate application and associated dependencies, as well as other components, to provide a complete file system, but shares the kernel functionality of the host operating system with other application containers. Each application container executes as an isolated process in the user space of the host operating system. Specifically, the container system utilizes an underlying operating system that uses virtual memory for isolation to support the provision of basic services to all containerized applications. One or more containers may be instantiated to execute one or more applications or functions of server node 500, and to execute one or more of the various modules and functions of the storage control system 150 discussed herein. In yet another embodiment, containers may be used in combination with other virtualization infrastructure such as virtual machines implemented using hypervisors, wherein Docker containers or other types of LXC are configured to run on virtual machines in a multi-tenant environment.

[0095] Various components of the storage control system, snapshot management system, and data backup management system include program code loaded into system memory 510 (e.g., volatile memory 512) and executed by processor 502 to perform the corresponding functions described herein. In this regard, system memory 510, storage resource 516, and other memory or storage resources described herein that tangibly contain program code and data thereon are examples of what is more generally referred to herein as a "processor-readable storage medium" storing executable program code for one or more software programs. Articles of manufacture including such processor-readable storage media are considered embodiments of this disclosure. Articles of manufacture may include, for example, storage devices such as storage disks, storage arrays, or integrated circuits containing memory. The term "article of manufacture" as used herein should be understood to exclude transiently propagating signals.

[0096] System memory 510 includes various types of memory, such as volatile RAM, NVRAM, or other types of memory, in any combination. Volatile memory 512 may be dynamic random access memory (DRAM) (e.g., DRAMDIMM (Dual In-line Memory Module)) or other forms of volatile RAM. Non-volatile memory 514 may include one or more of NAND flash memory devices, SSD devices, or other types of next-generation non-volatile memory (NGNVM) devices. System memory 510 can be implemented using a hierarchical memory layer structure, where volatile system memory 512 is configured as the highest-level memory layer, and non-volatile system memory 514 (and other additional non-volatile memory devices, including storage-class memory) is configured as a lower-level memory layer, which is used as a high-speed load / store non-volatile memory device on the processor memory bus (i.e., accessing data by load and store, rather than by I / O read and write). As used herein, the term "memory" or "system memory" refers to volatile and / or non-volatile memory used to store application instructions read and processed by processor 502 to execute the native operating system and one or more applications or processes hosted by server node 500, and to temporarily store data used and / or generated by the native OS and applications and processes running on server node 500. Storage resource 516 may include one or more HDDs, SSDs, etc.

[0097] It should be understood that the embodiments described above are presented for illustrative purposes only. Many variations can be made in the specific arrangements shown. For example, although described in the context of a particular system and device configuration, these techniques can be applied to a wide variety of other types of information processing systems, computing systems, data storage systems, processing devices, and distributed virtual infrastructure deployments. Furthermore, any simplifying assumptions made above in describing the illustrative embodiments should be considered exemplary and not as requirements or limitations of such embodiments. Many other alternative embodiments within the scope of the appended claims will be apparent to those skilled in the art.

Claims

1. A method comprising: The storage control system receives user input / output (I / O) write requests, which include data to be written to a RAID (Redundant Array of Independent Drives) array managed by the storage control system, the RAID array comprising multiple storage devices; The storage control system determines whether the RAID array is operating in a degraded mode, in which at least one of the plurality of storage devices in the RAID array is inaccessible due to a failure; as well as In response to determining that the RAID array is operating in degraded mode, determining whether to perform a degraded write operation to write the data to the RAID array in the degraded mode, wherein determining whether to perform the degraded write operation includes: The storage control system determines (i) whether spare capacity has been allocated to the RAID array for rebuilding lost data from at least one inaccessible storage device of the RAID array, and (ii) whether lost data blocks associated with the data of the I / O write request have been rebuilt into the allocated spare capacity. as well as In response to determining that (i) the spare capacity has been allocated to the RAID array, but the lost data block associated with the data of the I / O write request has not yet been reconstructed to the allocated spare capacity and (ii) the latency of performing a write operation on the spare capacity exceeds a latency threshold, the storage control system processes the user I / O write request by performing the degraded write operation using the remaining storage devices among the plurality of storage devices of the RAID array, without using the spare capacity.

2. The method of claim 1, wherein in response to determining that the RAID array is operating in degraded mode, the method further comprises: The storage control system determines whether the remaining resilience level of the RAID array is sufficient to provide fault tolerance for at least one additional failure of the remaining storage devices in the RAID array; In response to determining that the remaining resilience level of the RAID array is sufficient to provide fault tolerance for at least one additional failure of the remaining storage devices of the RAID array, the storage control system uses the remaining storage devices of the plurality of storage devices in the RAID array to perform the degradation write operation without using the allocated spare capacity; as well as In response to determining that the remaining resilience level of the RAID array is insufficient to provide fault tolerance for at least one additional failure of the remaining storage devices of the RAID array, the storage control system processes the user I / O write request by performing a write operation using the allocated spare capacity and the remaining storage devices of the RAID array.

3. The method of claim 2, wherein determining whether the remaining resilience level of the RAID array is sufficient to provide fault tolerance for at least one additional failure of the remaining storage devices of the RAID array comprises: The storage control system determines the number of currently inaccessible storage devices in the RAID array; In response to determining that the number of currently inaccessible storage devices is less than the maximum number of storage device failures that the RAID array is configured to handle to provide fault tolerance, the storage control system determines that the remaining resilience level of the RAID array is sufficient. as well as In response to determining that the number of currently inaccessible storage devices is equal to the maximum number of storage device failures that the RAID array is configured to handle to provide fault tolerance, the storage control system determines that the remaining resilience level of the RAID array is insufficient.

4. The method of claim 3, wherein the RAID array is configured to provide fault tolerance for failure of not less than two storage devices.

5. The method of claim 2, wherein determining whether the remaining resilience level of the RAID array is sufficient to provide fault tolerance for at least one additional failure of the remaining storage devices of the RAID array comprises: The storage control system determines the number of currently inaccessible storage devices in the RAID array; In response to determining that the number of currently inaccessible storage devices is less than a resilience threshold, the storage control system determines that the remaining resilience level of the RAID array is sufficient, wherein the resilience threshold is less than the maximum number of storage device failures that the RAID array is configured to handle to provide fault tolerance. as well as In response to determining that the number of currently inaccessible storage devices is equal to or greater than the resilience threshold, the storage control system determines that the remaining resilience level of the RAID array is insufficient.

6. The method of claim 1, wherein determining that the delay in performing the write operation on the spare capacity exceeds the delay threshold comprises: It is determined that the latency of performing the degradation write operation is less than the latency associated with waiting for the availability of the spare capacity to process the I / O write request.

7. The method of claim 1, wherein in response to determining that the RAID array is operating in a degraded mode, the method further comprises: The storage control system requests access to the allocated spare capacity to process the user I / O write request; as well as In response to a request to access the allocated spare capacity being served before the expiration of a specified time period, the storage control system uses the allocated spare capacity and the remaining storage devices among the plurality of storage devices in the RAID array to perform a write operation. When a request to access the allocated spare capacity is not served before the expiration of the specified time period, the storage control system uses the remaining storage devices among the plurality of storage devices in the RAID array to perform the degraded write operation without using the allocated spare capacity.

8. The method of claim 1, wherein the RAID array configuration includes a RAID level 1 configuration having fault tolerance for failure of at least two of the plurality of storage devices in the RAID array.

9. The method of claim 1, wherein the RAID array configuration includes a parity-based RAID configuration having fault tolerance for failure of at least two of the plurality of storage devices in the RAID array.

10. An article of manufacture comprising a non-transient processor-readable storage medium having program code of one or more software programs stored therein, wherein the program code is executable by the one or more processors to implement a method, the method comprising: The storage control system receives user input / output (I / O) write requests, which include data to be written to a RAID (Redundant Array of Independent Drives) array managed by the storage control system, the RAID array comprising multiple storage devices; The storage control system determines whether the RAID array is operating in a degraded mode, in which at least one of the plurality of storage devices in the RAID array is inaccessible due to a failure; as well as In response to determining that the RAID array is operating in degraded mode, determining whether to perform a degraded write operation to write the data to the RAID array in the degraded mode, wherein determining whether to perform the degraded write operation includes: The storage control system determines (i) whether spare capacity has been allocated to the RAID array for rebuilding lost data from at least one inaccessible storage device of the RAID array, and (ii) whether lost data blocks associated with the data of the I / O write request have been rebuilt into the allocated spare capacity. as well as In response to determining that (i) the spare capacity has been allocated to the RAID array, but the lost data block associated with the data of the I / O write request has not yet been reconstructed to the allocated spare capacity and (ii) the latency of performing a write operation on the spare capacity exceeds a latency threshold, the storage control system processes the user I / O write request by performing the degraded write operation using the remaining storage devices among the plurality of storage devices of the RAID array, without using the spare capacity.

11. The article of manufacture of claim 10, wherein in response to determining that the RAID array is operating in a degraded mode, the method further comprises: The storage control system determines whether the remaining resilience level of the RAID array is sufficient to provide fault tolerance for at least one additional failure of the remaining storage devices in the RAID array; In response to determining that the remaining resilience level of the RAID array is sufficient to provide fault tolerance for at least one additional failure of the remaining storage devices of the RAID array, the storage control system uses the remaining storage devices of the plurality of storage devices in the RAID array to perform the degradation write operation without using the allocated spare capacity; as well as In response to determining that the remaining resilience level of the RAID array is insufficient to provide fault tolerance for at least one additional failure of the remaining storage devices of the RAID array, the storage control system processes the user I / O write request by performing a write operation using the allocated spare capacity and the remaining storage devices of the RAID array.

12. The article of manufacture of claim 11, wherein determining whether the remaining resilience level of the RAID array is sufficient to provide fault tolerance for at least one additional failure of the remaining storage devices of the RAID array comprises: The storage control system determines the number of currently inaccessible storage devices in the RAID array; In response to determining that the number of currently inaccessible storage devices is less than the maximum number of storage device failures that the RAID array is configured to handle to provide fault tolerance, the storage control system determines that the remaining resilience level of the RAID array is sufficient. as well as In response to determining that the number of currently inaccessible storage devices is equal to the maximum number of storage device failures that the RAID array is configured to handle to provide fault tolerance, the storage control system determines that the remaining resilience level of the RAID array is insufficient.

13. The article of manufacture of claim 11, wherein determining whether the remaining resilience level of the RAID array is sufficient to provide fault tolerance for at least one additional failure of the remaining storage devices of the RAID array comprises: The storage control system determines the number of currently inaccessible storage devices in the RAID array; In response to determining that the number of currently inaccessible storage devices is less than a resilience threshold, the storage control system determines that the remaining resilience level of the RAID array is sufficient, wherein the resilience threshold is less than the maximum number of storage device failures that the RAID array is configured to handle to provide fault tolerance. as well as In response to determining that the number of currently inaccessible storage devices is equal to or greater than the resilience threshold, the storage control system determines that the remaining resilience level of the RAID array is insufficient.

14. The article of manufacture of claim 10, wherein determining that the delay in performing the write operation on the spare capacity exceeds the delay threshold comprises: It is determined that the latency of performing the degradation write operation is less than the latency associated with waiting for the availability of the spare capacity to process the I / O write request.

15. The article of manufacture of claim 10, wherein in response to determining that the RAID array is operating in a degraded mode, the method further comprises: The storage control system requests access to the allocated spare capacity to process the user I / O write request; as well as In response to a request to access the allocated spare capacity being served before the expiration of a specified time period, the storage control system uses the allocated spare capacity and the remaining storage devices among the plurality of storage devices in the RAID array to perform a write operation. When a request to access the allocated spare capacity is not served before the expiration of the specified time period, the storage control system uses the remaining storage devices among the plurality of storage devices in the RAID array to perform the degraded write operation without using the allocated spare capacity.

16. A system comprising: At least one processor; as well as A system memory is configured to store program code, wherein the program code is executable by the at least one processor to implement a storage control system, the storage control system being configured to: Receive user input / output (I / O) write requests, the user input / output (I / O) write requests including data to be written to a RAID (Redundant Array of Independent Drives) array managed by the storage control system, the RAID array including multiple storage devices; Determine whether the RAID array is operating in a degraded mode, in which at least one of the plurality of storage devices of the RAID array is inaccessible due to a failure; as well as In response to determining that the RAID array is operating in degraded mode, the storage control system is configured to determine whether to perform a degraded write operation to write the data to the RAID array in the degraded mode, wherein determining whether to perform the degraded write operation, the storage control system is configured to: Determine (i) whether the spare capacity has been allocated to the RAID array for rebuilding lost data from at least one inaccessible storage device of the RAID array, and (ii) whether the lost data block associated with the data of the I / O write request has been rebuilt to the allocated spare capacity; as well as In response to determining that (i) the spare capacity has been allocated to the RAID array, but the lost data block associated with the data of the I / O write request has not yet been reconstructed to the allocated spare capacity and (ii) the latency of performing a write operation on the spare capacity exceeds a latency threshold, the user I / O write request is processed by performing the degraded write operation using the remaining storage devices among the plurality of storage devices of the RAID array, without using the spare capacity.

17. The system of claim 16, wherein in response to determining that the RAID array is operating in a degraded mode, the storage control system is configured to: Determine whether the remaining resilience level of the RAID array is sufficient to provide fault tolerance for at least one additional failure of the remaining storage devices in the RAID array; In response to determining that the remaining resilience level of the RAID array is sufficient to provide fault tolerance for at least one additional failure of the remaining storage devices of the RAID array, the remaining storage devices of the plurality of storage devices in the RAID array are used to perform the degradation write operation without using the allocated spare capacity; as well as In response to determining that the remaining resilience level of the RAID array is insufficient to provide fault tolerance for at least one additional failure of the remaining storage devices of the RAID array, the user I / O write request is processed by performing a write operation using the allocated spare capacity and the remaining storage devices of the RAID array.

18. The system of claim 16, wherein, when determining whether the remaining resilience level of the RAID array is sufficient to provide fault tolerance for at least one additional failure of the remaining storage devices of the RAID array, the storage control system is configured to: Determine the number of currently inaccessible storage devices in the RAID array; In response to determining that the number of currently inaccessible storage devices is less than the maximum number of storage device failures that the RAID array is configured to handle to provide fault tolerance, it is determined that the remaining resilience level of the RAID array is sufficient. as well as In response to determining that the number of currently inaccessible storage devices is equal to the maximum number of storage device failures that the RAID array is configured to handle to provide fault tolerance, it is determined that the remaining resilience level of the RAID array is insufficient.

19. The system of claim 16, wherein determining that the delay in performing the write operation on the spare capacity exceeds the delay threshold comprises: It is determined that the latency of performing the degradation write operation is less than the latency associated with waiting for the availability of the spare capacity to process the I / O write request.

20. The system of claim 16, wherein in response to determining that the RAID array is operating in a degraded mode, the storage control system is configured to: Request access to the allocated spare capacity to process the user I / O write request; and In response to the request to access the allocated spare capacity being served before the expiration of a specified time period, a write operation is performed using the allocated spare capacity and the remaining storage devices among the plurality of storage devices in the RAID array; When a request to access the allocated spare capacity is not served before the expiration of the specified time period, the storage control system uses the remaining storage devices among the plurality of storage devices in the RAID array to perform the degraded write operation instead of using the allocated spare capacity.

Images