Method and system for recovery when first server is inaccessible

By distributing storage blocks between multiple servers and utilizing forward error correction solutions, the data loss and performance bottleneck problems of traditional distributed storage systems when a single server fails, and the rapid failure recovery of distributed storage systems is achieved and High availability.

CN119088618APending Publication Date: 2024-12-06WEKA IO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411176915.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-09-04
Filing Date
2018-10-05
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Traditional distributed storage systems cannot effectively scale when a single server fails, resulting in data loss and performance bottlenecks, and it is difficult for the existing technology to achieve rapid failure recovery.

Method used

By distributing storage blocks between multiple servers, the distributed fault elastic address space (DFRAS) and forward error correction scheme can realize the reconstruction and recovery of data blocks, ensuring that the system quickly recovers metadata and file systems in the event of server failure. usability.

Benefits of technology

It realizes scalability and high-performance storage for a large number of nodes, enables rapid data recovery in the event of a single server failure, reduces downtime, and improves system reliability and fault tolerance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088618A_ABST
    Figure CN119088618A_ABST
Patent Text Reader

Abstract

Methods and systems are provided for recovery when a first server is inaccessible. The method comprises the following steps: reconstructing a data block of a first server; and storing the reconstructed data blocks in a second server, in which: the data blocks in the first server are associated with a first plurality of stripes, the second plurality of stripes comprising the reconstructed data blocks, each stripe of the first plurality of stripes and the second plurality of stripes spanning a different set of servers, and each different server set comprises at least two servers.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the Chinese national phase application with an international filing date of October 5, 2018, an international application number of PCT / IB2018 / 001255, and an invention name of “Method and system for rapid fault recovery of distributed storage systems”. The national phase entry date of the Chinese national phase application is July 13, 2020, the application number is 201880086368.0, and the invention name is “Method and system for rapid fault recovery of distributed storage systems”.

[0002] Priority claim

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 585,166, filed on November 13, 2017, entitled “Method and System for Rapid Fault Recovery of Distributed Storage Systems” and U.S. Patent Application No. 16 / 121,491, filed on September 4, 2018, entitled “Method and System for Rapid Fault Recovery of Distributed Storage Systems”. Background Art

[0004] By comparing such an approach with some aspects of the present method and system set forth in the remainder of this disclosure with reference to the accompanying figures, the limitations and shortcomings of conventional approaches to data storage will become apparent to those skilled in the art.

[0005] CROSS-REFERENCE TO RELATED APPLICATIONS

[0006] The entirety of U.S. Patent Application No. 15 / 243,519, entitled "Distributed Erasure Coded Virtual File System," is incorporated herein by reference in its entirety. Summary of the invention

[0007] Methods and systems for fast failure recovery in a distributed storage system are provided, the methods and systems being substantially as illustrated by and / or described in conjunction with at least one of the accompanying drawings and more fully set forth in the claims.

[0008] On the one hand, the present application provides a method for recovery when a first server is inaccessible, comprising: reconstructing data blocks of the first server; and storing the reconstructed data blocks in a second server, wherein: the data blocks in the first server are associated with a first plurality of stripes, the second plurality of stripes include the reconstructed data blocks, each stripe in the first plurality of stripes and the second plurality of stripes spans a different set of servers, and each different set of servers includes at least two servers.

[0009] On the other hand, the present application provides a system comprising multiple storage devices controlled by multiple servers, the system comprising: one or more processors configured to: determine that a first server is inaccessible, rebuild a data block of the first server, determine a second server to replace the first server, and store the reconstructed data block in the second server, wherein: the data block in the first server is associated with a first plurality of stripes, the second plurality of stripes include the reconstructed data block, each stripe in the first plurality of stripes and the second plurality of stripes spans a different set of servers, and each different set of servers includes at least two servers. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 Various example configurations of a virtual file system according to aspects of the present disclosure are shown.

[0011] Figure 2 An example configuration of a virtual file system node according to aspects of the present disclosure is shown.

[0012] Figure 3 Another representation of a virtual file system according to an example implementation of the present disclosure is shown.

[0013] Figure 4 A simplified example showing the relationship between the number of nodes and the number of quintuples according to an example implementation of the present disclosure.

[0014] Figure 5 A flow chart illustrating an example method of recovering from a single node failure according to an example implementation of the present disclosure.

[0015] Figure 6 A flow chart illustrating an example method of recovering from a multi-node failure according to an example implementation of the present disclosure.

[0016] Figure 7 An example implementation is shown in which two distributed failure-resilient address spaces reside on multiple solid-state storage disks.

[0017] Figure 8 A forward error correction scheme according to an example implementation of the present disclosure is shown, which may be used to protect data stored to a non-volatile memory of a virtual file system. DETAILED DESCRIPTION

[0018] Traditionally, file systems use centralized control of metadata structures (e.g., directories, files, attributes, file contents). If a local file system is accessible from a single server and that server fails, the data for that file system may be lost without further protection. To increase protection, some file systems (e.g., those provided by NetApp) use one or more pairs of controllers in an active-passive fashion to replicate metadata across two or more machines. Other solutions use multiple metadata servers in a clustered fashion (e.g., provided by IBM GPFS, Dell EMC Isilon, Lustre, etc.). However, because the number of metadata servers in a traditional clustered system is limited to a small number, such systems cannot scale.

[0019] The system in the present disclosure is suitable for small clusters and can also be expanded to many tens of thousands of nodes. Example embodiments of non-volatile memory (NVM) (e.g., flash memory in the form of a solid-state drive (SSD)) are discussed. NVM can be divided into 4kB "blocks" and 128MB "chunks". "Extents" can also be stored in volatile memory, such as RAM for fast access, or can be backed up by NVM storage. The extension area can store pointers to blocks, for example, 256 pointers to 1MB of data stored in the block. In other embodiments, larger or smaller memory partitions can also be used. The metadata functions in the present disclosure can be effectively distributed on many servers. For example, in the case of a "hot spot" that points to a specific part of the file system namespace, this load can be distributed on multiple nodes.

[0020] Figure 1 Various example configurations of a virtual file system (VFS) according to aspects of the present disclosure are shown. Figure 1 Shown in the figure is a local area network (LAN) 102, which includes one or more VFS nodes 120 (indexed by integers from 1 to J, j ≥ 1), and optionally includes (shown in dashed lines): one or more dedicated storage nodes 106 (indexed by integers from 1 to M, M ≥ 1); one or more compute nodes 104 (indexed by integers from 1 to N, N ≥ 1); and / or an edge router connecting the LAN 102 to a remote network 118. The remote network 118 optionally includes one or more storage services 114 (indexed by integers from 1 to K, K ≥ 1); and / or one or more dedicated storage nodes 115 (indexed by integers from 1 to L, for L ≥ 1).

[0021] 120 per VFS node j(j is an integer, where 1≤j≤J) is a network computing device (e.g., a server, a personal computer, etc.) that includes circuitry for running a VFS process and an optional client process (or directly on device 104). n operating system and / or on the device 104 n in one or more virtual machines running in a .

[0022] Compute node 104 is a networking device that can run a VFS front end without a VFS back end. Compute node 104 can run the VFS front end by putting SR-IOV into the NIC and using a full processor core. Alternatively, compute node 104 can run the VFS front end by routing the networking via the Linux kernel networking stack and using kernel process scheduling, thus not having the requirement of a full kernel. This is useful if the user does not want to allocate a full core for VFS, or the networking hardware is not compatible with the VFS requirements.

[0023] Figure 2 An example configuration of a VFS node according to aspects of the present disclosure is shown. The VFS node includes a VFS front end 202 and a driver 208, a VFS memory controller 204, a VFS back end 206, and a VFS SSD agent 214. As used in the present disclosure, a "VFS process" is a process that implements one or more of the following: a VFS front end 202, a VFS memory controller 204, a VFS back end 206, and a VFS SSD agent 214. Therefore, in an example implementation, resources (e.g., processing and memory resources) of a VFS node can be shared between client processes and VFS processes. The processes of the VFS can be configured to require a relatively small amount of resources to minimize the impact on the performance of client applications. The VFS front end 202, the VFS memory controller 204, and / or the VFS back end 206, and / or the VFS SSD agent 214 can run on a processor of the host 201 or on a processor of a network adapter 218. For a multi-core processor, different VFS processes can run on different cores and can run different subsets of services. From the perspective of the client process 212, the interface with the virtual file system is independent of the specific physical machine running the VFS process. The client process only needs the driver 208 and the front end 202 to exist and can provide services for them.

[0024] A VFS node can be implemented as a single tenant server (e.g., bare metal) running directly on an operating system, or as a virtual machine (VM) and / or container (e.g., Linux container (LXC)) in a bare metal server. VFS can run as a VM environment in an LXC container. Therefore, inside the VM, only an LXC container containing the VFS can run. In a classic bare metal environment, there are user space applications and the VFS runs in an LXC container. If the server is running other containerized applications, the VFS may run in an LXC container outside the management scope of the container deployment environment (e.g., Docker).

[0025] The VFS node can be served by an operating system and / or a virtual machine monitor (VMM) (e.g., a hypervisor). The VMM can be used to create and run a VFS node on a host 201. Multiple kernels can reside in a single LXC container running a VFS, and the VFS can run on a single host 201 using a single Linux kernel. Therefore, a single host 201 may include multiple VFS front ends 202, multiple VFS memory controllers 204, multiple VFS back ends 206, and / or one or more VFS drivers 208. The VFS driver 208 can run in kernel space outside the scope of the LXC container.

[0026] A single root input / output virtualization (SR-IOV) PCIe virtual function can be used to run a network stack 210 in user space 222. SR-IOV allows isolation of PCI Express so that a single physical PCI Express can be shared on a virtual environment, and different virtual functions can be provided to different virtual components on a single physical server computer. The I / O stack 210 enables the VFS node to bypass the standard TCP / IP stack 220 and communicate directly with the network adapter 218. A portable operating system interface for uniX (POSIX) VFS functions can be provided to the VFS driver 208 through a lock-free queue. SR-IOV or a full PCIe physical function address can also be used to run a non-volatile memory Express (NVMe) driver 214 in user space 222, completely bypassing the Linux IO stack. NVMe can be used to access non-volatile storage media 216 connected via a PCI Express (PCIe) bus. The non-volatile storage medium 216 may be, for example, flash memory in the form of a solid state drive (SSD) or a storage class memory (SCM) in the form of a solid state drive (SSD) or a memory module (DIMM). Other examples may include storage class memory technologies such as 3D-XPoint.

[0027] By coupling the physical SSD 216 with the SSD agent 214 and the networking 210, the SSD can be implemented as a networked device. Alternatively, the SSD can be implemented as a network-attached NVMe SSD 242 or 244 by using a network protocol such as NVMe-oF (NVMe over fabric). NVMe-oF can allow access to NVMe devices using redundant network links, thereby providing a higher level or resilience. Network adapters 226, 228, 230, and 232 can include hardware acceleration for connecting to NVMe SSDs 242 and 244 to convert them into networked NVMe-oF devices without the use of a server. NVMe SSDs 242 and 244 can each include two physical ports, and all data can be accessed through any one of these ports.

[0028] Each client process / application 212 can run directly on the operating system, or can run in a virtual machine and / or container served by the operating system and / or hypervisor. The client process 212 can read data from the memory and / or write data to the memory in the process of performing its main function. However, the main function of the client process 212 is independent of storage (that is, the process only focuses on its data being reliably stored and retrievable when needed, without paying attention to the storage location, time or method of the data). Example applications that cause this process include: email servers, web servers, office efficiency applications, customer relationship management (CRM), animation video rendering, genome computing, chip design, software construction, and enterprise resource planning (ERP).

[0029] The client application 212 can make a system call to the kernel 224 that communicates with the VFS driver 208. The VFS driver 208 places the corresponding request on the queue of the VFS front end 202. If there are several VFS front ends, the driver may load-balance access to different front ends to ensure that a single file / directory is always accessed through the same front end. This can be done by "sharding" the front end based on the ID of the file or directory. The VFS front end 202 provides an interface for routing file system requests to the appropriate VFS back end based on the storage bucket responsible for the operation. The appropriate VFS back end can be on the same host or on another host.

[0030] The VFS backend 206 hosts multiple buckets, each of which services file system requests it receives and performs tasks to otherwise manage the virtual file system (e.g., load balancing, logging, maintaining metadata, caching, moving data between tiers, deleting obsolete data, correcting corrupted data, etc.).

[0031] The VFS SSD agent 214 handles the interaction with the corresponding storage device 216. This may include, for example, translating addresses, and generating commands issued to the storage device (e.g., on a SATA, SAS, PCIe, or other suitable bus). Thus, the VFS SSD agent 214 acts as an intermediary between the VFS backend 206 of the virtual file system and the storage device 216. The SSD agent 214 may also communicate with standard network storage devices that support standard protocols, such as NVMe-oF (NVMe over Fabric).

[0032] Figure 3 Another representation of a virtual file system according to an example implementation of the present disclosure is shown. Figure 3 , element 302 represents the memory resources (e.g., DRAM and / or other short-term memory) and processing (e.g., x86 processor, ARM processor, NIC, ASIC) resources of the respective nodes (compute, storage and / or VFS) on which the virtual file system resides, for example, as described above with respect to Figure 2 Element 308 represents one or more physical storage devices 216 that provide long-term storage for the virtual file system.

[0033] like Figure 3 As shown, physical storage is organized into multiple distributed failure resilience address spaces (DFRAS) 518. Each distributed failure resilience address space includes multiple groups of blocks 310, which in turn include multiple blocks 312. The organization of blocks 312 into groups of blocks 310 is only a convenience in some implementations and may not be performed in all implementations. Each block 312 stores committed data 316 (which can take various states discussed below) and / or metadata 314 that describes or references the committed data 316.

[0034] Organizing storage device 308 into multiple DFRAS enables high performance parallel commits (e.g., Figure 1 All nodes 1041–104 N , 1061–106 M and 1201–120 J Concurrent commits may be performed in parallel.) In one example implementation, each node of the virtual file system may own one or more of the multiple DFRASs and have exclusive read / commit access rights to the DFRASs it owns.

[0035] Each bucket has its own DFRAS and therefore does not need to coordinate with any other node when writing to it. Each bucket may be striped across many different chunks on many different SSDs, so each bucket and its DFRAS can choose the current "chunk stripe" to write based on many parameters, and once a chunk is assigned to that bucket, it can be written without coordination. All buckets can effectively write to all SSDs without coordination.

[0036] Each DFRAS is owned and accessed only by its owner bucket running on a particular node, thereby allowing each node of the VFS to control a portion of the storage device 308 without having to coordinate with any other node (except during initialization or during [reallocation] of buckets holding DFRAS after a node failure, which may be performed asynchronously with actual read / commit operations on the storage device 308, for example). Thus, in such an implementation, each node may read / commit DFRAS to its bucket independently of what other nodes are doing, without requiring any consensus to be reached when reading and committing to the storage device 308. Furthermore, in the event of a failure of a particular node, the fact that the particular node owns multiple buckets allows for a smarter and more efficient reallocation of its workload to other nodes (rather than assigning the entire workload to a single node, potentially creating a "hotspot"). In this regard, in some implementations, the number of buckets may be large relative to the number of nodes in the system, such that any one bucket may be a relatively small load placed on another node. This allows for fine-grained redistribution of the load of a failed node based on the power and capacity of other nodes (e.g., nodes with more power and capacity may receive a higher percentage of the failed node's bucket).

[0037] To allow this operation, metadata may be maintained that maps each bucket to its currently owning node so that reads and commits to storage device 308 may be redirected to the appropriate node.

[0038] Load distribution is possible because the entire file system metadata space (e.g., directories, file attributes, ranges of content within files, etc.) can be broken down (e.g., chopped or fragmented) into uniform small pieces (e.g., "shards"). For example, a large system with 30k servers can divide the metadata space into 128k or 256k fragments.

[0039] Each such metadata shard can be stored in a "bucket". Each VFS node may be responsible for several buckets. When a bucket is serving metadata shards on a given backend, the bucket is considered "active" or "leader" for that bucket. Typically, there are more buckets than VFS nodes. For example, a small system with 6 nodes can have 120 buckets, while a large system with 1,000 nodes can have 8k buckets.

[0040] Each bucket can be active on a small group of nodes, typically 5 nodes that make up the quintuple for that bucket. The cluster configuration keeps all participating nodes up to date with the quintuple assignment for each bucket.

[0041] Each quintuple monitors itself. For example, if there are 10k servers in the cluster, and each server has 6 buckets, each server will only need to communicate with 30 different servers to maintain the state of its bucket (6 buckets will have 6 quintuples, so 6*5=30). This is much less than a centralized entity having to monitor all nodes and maintain cluster-wide state. Using quintuples allows performance to scale with larger clusters, because nodes don't do more work when the cluster size increases. This can have the disadvantage that in "dumb" mode a small cluster can actually generate more communication than physical nodes, but this can be overcome by only sending a single heartbeat between two servers because they share all buckets (as the cluster grows, this will change to just one bucket, but if you have a small cluster of 5 servers, it will just include all buckets in all messages, and each server will only communicate with 4 other servers). Quintuples can make decisions (i.e. reach consensus) using an algorithm similar to the Raft consensus algorithm.

[0042] Each bucket may have a set of compute nodes that can run it. For example, five VFS nodes can run a bucket. However, at any given moment, only one node in the group is the controller / leader. Furthermore, for a large enough cluster, no two buckets share the same group. If there are only 5 or 6 nodes in the cluster, most buckets can share a backend. In a fairly large cluster, there may be many different node groups. For example, in the case of 26 nodes, there are more than 64,000 possible five-node groups (i.e., quintuples).

[0043] All nodes in the group know and agree (i.e., reach consensus) which node is the actual active controller (i.e., leader) for that bucket. A node accessing a bucket may remember ("cache") the last node that was the leader for that bucket among the (e.g., five) members in the group. If it accesses the bucket leader, the bucket leader performs the requested operation. If the node it accesses is not the current leader, the node instructs the leader to "redirect" the access. If access to the cached leader node times out, the contacting node can try to use another node for the same five-tuple. All nodes in the cluster share a common "configuration" of the cluster, which allows the node to know which server can run each bucket.

[0044] Each bucket can have a load / utilization value that indicates how heavily the bucket is being used by applications running on the file system. For example, a server node with 11 buckets with low utilization can receive another metadata bucket to run before a server with 9 buckets with high utilization, even if the number of buckets used is unbalanced. The load value can be determined based on average response latency, number of operations running concurrently, memory consumption, or other metrics.

[0045] Redistribution may occur even if a VFS node is not faulty. If the system identifies that one node is busier than the others based on tracked load metrics, the system may move (i.e., "failover") one of its buckets to another, less busy server. However, before actually relocating the bucket to a different host, load balancing may be achieved by shifting writes and reads. Since each write may end up on a different set of nodes as determined by DFRAS, a node with a higher load may not be chosen in the stripe to which the data is written. The system may also choose not to serve reads from a highly loaded node. For example, a "degraded mode read" may be performed, in which blocks in a highly loaded node are reconstructed from other blocks in the same stripe. A degraded mode read is a read performed by the remaining nodes in the same stripe, and the data is reconstructed with failover. A degraded mode read may be performed when read latency is too high, because the initiator of the read may think that the node is down. If the load is high enough to create higher read latency, the cluster may revert to reading that data from other nodes and using degraded mode reads to reconstruct the required data.

[0046] Each bucket manages its own distributed erasure coding instance (i.e., DFRAS 518) and does not need to coordinate with other buckets to perform read or write operations. There may be thousands of concurrent distributed erasure coding instances working at the same time, each for a different bucket. This is integral to scaling performance because it effectively divides any large file system into independent parts that do not require coordination, thereby providing high performance regardless of the scale.

[0047] Each bucket handles all file system operations that belong to its shard. For example, directory structures, file attributes, and file data ranges will fall under the jurisdiction of a specific bucket.

[0048] An operation done from any frontend starts by finding out which bucket owns the operation. The backend leader and node for that bucket is then determined. This determination can be performed by trying the most recently known leader. If the most recently known leader is not the current leader, then the node probably knows which node is the current leader. If the most recently known leader is no longer part of the bucket's 5-tuple, then that backend will let the frontend know that it should go back into the configuration to find the members of the bucket's 5-tuple. The distribution of operations allows complex operations to be handled by multiple servers instead of a single computer in a standard system.

[0049] If the cluster size is small (e.g., 5), and quintuples are used, there will be multiple buckets sharing the same group. As the cluster size grows, the buckets will be redistributed so that no two groups are the same.

[0050] Protection mechanisms can be used to protect data from occasional server failures. Depending on the level of protection, the protection mechanism may be able to withstand one or more server failures to enable fast recovery. Some examples of recovery procedures from server failures are described below. Common procedures may involve, for example, rebuilding only data that is not read cached and rebuilding the least protected data segments first. One aspect of recovery from a failure may involve efficiently rebuilding the data and allowing I / O to run while the server is out of service. Another aspect of recovery from a failure may involve quickly restoring metadata ownership so that writes can continue using the bucket.

[0051] When protecting in +2 mode, data can be recovered even if two servers are in concurrent failure mode. When protecting in +4 mode, up to four concurrent servers can be in concurrent failure mode while still allowing full recovery. When a failure is detected, the stripe with the most affected blocks is repaired, followed by the next most affected stripe, and so on. While the most affected stripe is being repaired, the system may be able to tolerate another server failure, at which point the currently most affected stripe has been repaired. Therefore, the quick response to repair stripes can minimize downtime for the affected servers. In addition, because the data and the metadata to access the data are distributed across all available servers, most systems are still available for read and write operations.

[0052] Figure 4 A simplified example of the relationship between the number of nodes and the number of 5-tuples according to an example implementation of the present disclosure is shown. Figure 4 , shows a simplified example system 400 including seven servers 401-413, where each server has a single SSD with eight blocks. Also shown are three stripes across five servers, where a five-server stripe may be referred to as a quintuple.

[0053] exist Figure 4 As can be seen in Figure 2, on any given server, there is a lot of overlap in the 5-tuples (stripes). For the three-strip example, six servers have two stripes, and one server has three stripes. Figure 4 For the example of seven nodes described in , there are 21 different quintuples. The calculation is as above: number of quintuples = [(node)! / ((5!)*(node–5))]. However, increasing the number of nodes to 28 increases the number of possible quintuples to over 98,000.

[0054] Therefore, by increasing from one SSD per server (7 nodes) to 4 SSDs per server (28 nodes), the number of quintuples increases from 21 to 98,000. Therefore, when stripes are assigned to quintuples to minimize overlap between servers’ quintuples, it can be seen that the likelihood of a large number of stripes being affected by a single server is very small.

[0055] Moreover, the four SSDs of each server can be placed independently of the server and each other so that at least for power and / or network access, when one SSD is inaccessible due to power outage or network access interruption, the other three SSDs are not affected. Even if one server fails and four SSDs are inaccessible, there are still more than 42,000 quintuples available. Therefore, it can be seen that as the number of SSDs controlled by a server increases, the number of quintuples affected by the decline of a single SSD decreases dramatically.

[0056] Therefore, when a server goes down, there may be some quintuples that are affected because one of their chunks is located on that server. When the second server fails, so that there are two failed servers at the same time, the number of quintuples affected by both servers will be much smaller than the number of quintuples affected by one server. Similarly, when there are three failed servers, the number of quintuples affected by all three servers is much smaller than the number of quintuples affected by any two of the three failed servers. The following are some example scenarios for recovering from failed servers.

[0057] Figure 5 A flowchart of an example method for recovering from a single node failure according to an example implementation of the present disclosure is shown. Figure 5 , a flow chart 500 is shown. At block 502, the system can access data of a server. At block 504, after an attempt to write to a failed server, the system can detect that the server is inaccessible. The server itself may be down, or the path to the server may be damaged.

[0058] At frame 506, all write operations to the inaccessible server will be stopped, and the system will be set to transfer the write operation to other servers associated with the stripe to be written. At frame 508, the system can automatically perform the degraded mode reading process by reading all stripe data blocks in parallel from other servers to recreate the lost block. Therefore, even if there is a request for the data of the lost block in the inaccessible server, the reduction of data access performance may also be very small. The operation of frame 506 and 508 is shown as being substantially parallel. However, various embodiments can be implemented as shown in the figure or one frame is prioritized over another frame.

[0059] At block 510, a check may be performed to see if the inaccessible server has been brought back online. For example, a message may be received from the previously inaccessible server that it is online and ready for read / write requests. If so, the system may proceed to block 514 to rebuild any data needed to bring the previously inaccessible server up to date. For example, this may require moving data that was written to other servers because the previously inaccessible server was inaccessible to that server.

[0060] If the inaccessible server has not been brought back online, the system may check at block 512 to see if a wait time has passed since the inaccessible server was detected. Because the system is designed to run on multiple application servers where there is redundancy protection against multiple outages, it may not be necessary to immediately begin the rebuilding process for the lost block. The inaccessible server may be unavailable for a predetermined wait time, such as 15 minutes. During this time, writes continue to be transferred to other servers and reads are performed from the recreated blocks.

[0061] If the server is still inaccessible after the wait time, then at block 514, the data in the inaccessible server may be rebuilt and moved to other servers. The wait time may be to allow for temporary downtime. For example, the outage may be due to a power failure, and the server may have recovered. The wait time may be changed based on various parameters, such as the amount of data access to / from the server and / or stripe, known outages external to the system, etc.

[0062] Once the previously inaccessible server is back online at 510, the data and metadata including the registry and logs can be used to read and verify that the data stored on that server is valid. Therefore, there is no need to rebuild valid data. Data that has been moved to other servers can be moved back to the current data or left as is. This may depend on the design and / or implementation.

[0063] Therefore, unlike other current storage systems, it is possible that in most cases, when a server goes down, there is almost no need to perform any additional data reconstruction. However, in rare cases, the data stored in the NVM may have corrupted data.

[0064] In order to rebuild the data when only one server is down, the affected data from the down server can be moved to other servers. Since the data is referenced by metadata in the bucket, the bucket originally located on that server may have been migrated to other servers to support metadata functions very quickly. For example, the migration may be much faster than the waiting time (e.g., 15 minutes). The migration time can be, for example, a matter of seconds. This can be performed by each bucket, and each bucket can have a distributed erasure coding system (DECS) that can traverse all stripes it manages to find the portion of the stripe stored on the down server. The damaged stripe portion of the bucket can then be rebuilt from all remaining servers, and reconstruction can be performed simultaneously for all stripes affected by the down server.

[0065] Because servers may have limits on the networking and local SSD throughput available to the rebuild process, the amount of rebuild can be minimized so that the server that is rebuilding the lost data can perform optimally. Thus, only the portion of the stripes that are actually stored on the downed server are rebuilt. If no data is stored in the block because it is free or in the object store read cache, the block may be marked as unused and does not need to be rebuilt. As mentioned above, data in the read cache may have been rebuilt during the wait time.

[0066] Once all DECS have rebuilt all stripes affected by the downed server, the rebuild is complete. Therefore, during the rebuild process, another available server will be selected to join the stripe, the lost block will be recalculated, and then the block can be written to the new server to complete the stripe. All other blocks of the bucket do not need to be affected.

[0067] Figure 6 A flow chart showing an example method of recovering from a multi-node failure according to an example implementation of the present disclosure. Failures of multiple servers may be handled preferentially for different levels of affected stripes. For ease of illustration, +2 protection will be described below. It should be noted that +4 or +n, where n>1, may be handled in a similar manner.

[0068] Reference Figure 6 , a flowchart 600 is shown, which can be substantially similar to flowchart 500. Block 602 can be similar to block 502. At block 604, a plurality of server failures can be determined. Blocks 606 and 608 can be similar to blocks 506 and 508.

[0069] For the sake of brevity, flowchart 600 omits blocks 510 and 512, but it should be understood that similar operations occur in flowchart 600. At block 610, the affected stripes are identified with the most affected stripe having a higher priority, then the next most affected stripe, and so on. For example, assuming two servers fail, when the wait time ends and the servers begin to be rebuilt, all DECSs can check their stripes to find the stripes affected by the two failed servers.

[0070] At block 612, those identified stripes are rebuilt first. This is because if there is another server failure that affects one of these stripes, the blocks on those servers may not be recoverable. After rebuilding the most affected stripe, rebuild the next most affected stripe, and so on.

[0071] As mentioned previously, stripes are spread across all servers to minimize the number of stripes that depend on the same server or group of servers. Therefore, the number of stripes affected by the failure of two servers may be smaller than the number of stripes affected by the failure of one of the servers. Therefore, rather than trying to rebuild every affected stripe, priority is given to the fewer stripes affected by two failed servers. As before, stripes are affected if the data blocks in the server actually hold data rather than being idle or copied to the read cache.

[0072] After all stripes with two actual failures are repaired, the system can then proceed to rebuild those stripes that were affected by only one of the two failed servers. The two different single server failures can be rebuilt in parallel or in series. This can depend on, for example, the design and / or implementation of the system. For example, if it is determined that network traffic is very high overall and / or to servers for buckets affected by the failed servers, it can be determined that one failed server can be rebuilt at a time.

[0073] If a third server fails while both servers are still down and have not been replaced, all DECS checks all stripes and identifies the stripes affected by the two failed servers. For example, this could be any stripes affected by the first failed server and the second failed server, the second failed server and the third failed server, or the first failed server and the third failed server. These identified stripes can be rebuilt first, then they are rebuilt, and then those stripes affected by the single failed server are rebuilt. Therefore, although three servers may fail, the system may not actually be compromised.

[0074] If one of the failed servers comes back online, it may not be necessary to continue rebuilding the data stored on that server. However, whether to rebuild may depend on several criteria. For example, the rebuild may depend on how far the rebuild has progressed. If the rebuild is almost complete to a certain extent, which may be predetermined and / or variable depending on recent conditions (e.g., network traffic, local traffic, etc.), the rebuild may continue. The rebuild may also depend on, for example, the history of server downtime and / or storage node downtime. If there is a problem with a particular server, a portion of a server, or a network path to a server, it may be more convenient to use a new server until the problem affecting the server that recently failed is resolved.

[0075] When having +4 protection, the same thing happens for the first, second, third, and fourth failures, and only the fourth failure can be dangerous. If there are stripes affected by all four servers that are affected by the fifth failed server, those stripes affected by the fifth failed server may not be able to be rebuilt. However, this example architecture can mitigate the possibility of such failures by having various buckets use different servers to reduce the number of stripes affected by the failed server. Therefore, when there are few stripes affected by four failures, all DECS can repair the stripe with 4 holes in a few seconds or minutes, so that the fifth server failure can occur very quickly and the system may still remain resilient.

[0076] To rebuild the stripe, the metadata in the affected buckets may be recovered first. Although the reconstruction process disclosed above describes the reconstruction process, for example, 15 minutes after the failure occurs, the bucket recovery process may need to proceed faster because all read / write operations to the file range (or directory operations, etc.) processed by the affected buckets may be stalled until each affected bucket has a leader.

[0077] The system can determine how many buckets were running on the failed server and determine the best server to accept those buckets from a load balancing perspective. If the system is large enough, only one bucket needs to be moved to each receiving server, but on smaller systems, a server may need to receive multiple buckets.

[0078] The system can track the activity window for each bucket and can try to assign buckets to less busy servers, or if buckets are grouped together, group buckets that are currently less loaded. This can alleviate the need to perform load balancing shortly after a rebuild.

[0079] The receiving server can perform processes to bring the bucket up to date in storage as quickly as possible so that the server can begin servicing requests. For example, this may require merging registries and logs to obtain coherent operations that work well during rebuilding of failed servers and / or load balancing.

[0080] The process may include delaying writing registry data to the NVM. This may be, for example, to reduce system delays due to write latency when storing registry data, or to reduce the number of writes to the NVM when the NVM may be limited in the number of writes it may allow during its lifetime. Thus, the registry may be written to the NVM after, for example, hundreds or thousands of operations. A log may be used to track operations between writes to store the registry to the NVM, where the registry may point to the current head of the corresponding log.

[0081] As data is added to the file, the log may grow, thus taking up more memory. Therefore, after a certain point, the information in the log can be "pushed" to the registry. Therefore, once the registry is stored to NVM, the memory occupied by the corresponding log can be freed.

[0082] So the logs can be used to update the registry. But they can also be used when a server / node fails due to power failure or other reasons, and when rebalancing the load by sending buckets to other backend servers.

[0083] In the event of a failure, the receiving server may be freed from responsibility for the bucket until the bucket is fully up and running. For metadata segments managed by the bucket at the receiving server, this may provide better performance during periods of reduced performance at the receiving server, for example.

[0084] Often, it may be necessary to quickly recover to full performance after a server failure or after a server power failure. As part of the recovery, the register level 1 (L1) cache and L1 replica registers may be loaded, then the level 2 (L2) cache and L2 replica registers may be loaded, and then the log may be found for the bucket. The log may now be replayed so that the individual registers may be updated. For example, the L1 cache and corresponding replica registers may be updated, then the L2 cache and corresponding replica registers may be updated as needed, and then the registers in the NVM may be updated as needed.

[0085] Although specific processes are described, various aspects of the present disclosure may use different processes to update the NVM register and the cache register. In addition, there may be different numbers of cache levels and / or there may be no copy registers.

[0086] The log can also be idempotent in that each replay of the log makes the same changes to the registry. Therefore, the log can be tested after it is created to verify that the registry is not changed after the first replay of the log. Therefore, it should also be noted that replaying log entries does not change or create log entries.

[0087] Additionally, since there may be many buckets with corresponding logs, replaying the logs sequentially may take a long time. Therefore, the replay of the logs can be optimized by determining the temporal relationship between the logs. Therefore, each log can also define the resources to be serialized. The process can then first perform the "scheduling" part of the replay to check which resources are serialized (or blocked) by each log entry, and then the schedule can be optimized for the resources to topologically sort the logs. This can then allow as many logs as possible to be replayed in parallel to reduce the time it takes to replay all the logs.

[0088] The bucket may not be set to ready until all logs have been successfully played. As long as the bucket is not ready, it will not accept new IO and / or metadata operations. IO that arrives during this period may receive a return code indicating that the bucket is not ready. Therefore, the responsibility for retries and / or other options may lie with the frontend sending the IO. This frees up the backend so that it does not have to keep track of IO requests while it is initializing. This can mitigate any issues that may occur if the system is very large and all frontends are constantly sending new IO to the bucket being initialized.

[0089] Once all log entries have been replayed, the new registry state can be saved to NVM and then all log entries can be released. The bucket can then be set up to be ready to handle IO.

[0090] Although the faulty server has been described Figure 5 and Figure 6 , but it should be noted that a server is specified for ease of description. In general, the various embodiments described can be used for any node that affects storage. For example, a node can be a server that controls many storage devices, such as SSD disks, etc.

[0091] Various aspects of the present disclosure may provide a method for recovering from a server failure, including determining that a first server is inaccessible. When it is determined that the first server is inaccessible, a degraded mode read may be performed for a read request for data in the inaccessible first server, and a write request to the inaccessible first server may be redirected to another server. It may be further determined whether a wait time has elapsed since the determination that the first server is inaccessible, wherein the wait time begins when the determination that the first server is inaccessible. If so, the data blocks in the first server may be rebuilt, a second server may be selected to replace the inaccessible first server, and the rebuilt data blocks may be moved to the second server. It should be noted that these steps may not need to be performed sequentially. For example, the second server may be determined before, during, or after the time period for rebuilding the data blocks.

[0092] If write access to the first server fails, it can be determined that the first server is inaccessible. The degraded mode read can include, for example, rebuilding the first data block in the inaccessible first server using stripe data blocks associated with the first data block, wherein each stripe data block is in a unique server different from the first server, and the reconstructed first data block can be populated in the read cache. Stripe blocks from different servers can be read in parallel. Read requests for the first data block can be serviced by the read cache. Redirecting the write request can include redirecting the write request to other servers associated with the data block targeted by the write request. These other servers may have blocks in the same stripe as the data block targeted by the write request. In the event that the wait time has not passed, the degraded mode read may continue to be executed; write requests to the inaccessible first server may continue to be redirected to another server.

[0093] Rebuilding the data blocks in the first server may include, for each data block in the first server, rebuilding each data block in the first server using stripe data blocks associated with each data block in the inaccessible first server, and the stripe data blocks may be read in parallel.

[0094] When the first server is one of the inaccessible multiple servers, the data blocks to be rebuilt can be prioritized. The prioritization can include, for example, first rebuilding one or more data blocks that are most affected by the inaccessible multiple servers.

[0095] The method may further include, for example, receiving notification from the first server that it is now accessible, determining whether the data in the first server is not corrupted, and when it is determined that the data is not corrupted, moving any data blocks that have been written to any server due to the first server being inaccessible to the first server.

[0096] Various aspects of the present disclosure may further provide a system comprising a plurality of storage devices controlled by a plurality of servers, and may include one or more processors configured to determine that a first server is inaccessible. When it is determined that the first server is inaccessible, the one or more processors may be configured to set the system to a degraded mode read for data in the first server, and set the system to redirect write requests to the first server to other servers. The one or more processors may be configured to determine whether a wait time has passed, wherein the wait time begins with a determination that the first server is inaccessible, and if so, then the one or more processors may be configured to rebuild data blocks in the first server, determine a second server to replace the first server, and then move the rebuilt data blocks to the second server. If write access to the first server fails, it may be determined that the first server is inaccessible.

[0097] The degraded mode read may include: reconstructing, by one or more processors, the first data block in the inaccessible first server using stripe data blocks associated with the first data block, wherein each stripe data block is located in a unique server different from the first server, and populating, by one or more processors, the reconstructed first data block in a read cache. The stripe data blocks may, for example, be read in parallel.

[0098] Redirecting the write request may include redirecting the write request to another server associated with the data block targeted by the write request. Rebuilding the data blocks in the inaccessible first server may include, for example, for each data block in the first server, reading, by one or more processors, stripe data blocks associated with each data block in the first server to rebuild each data block in the first server.

[0099] When the first server is one of the inaccessible multiple servers, the one or more processors may prioritize data blocks on the first server to be rebuilt by determining one or more data blocks most affected by the inaccessible multiple servers.

[0100] The one or more processors may be configured to receive a notification from the first server that the first server is online, determine whether data in the first server is not corrupted, and when it is determined that the data is not corrupted, move any data blocks that have been written to another server due to the first server being inaccessible to the first server.

[0101] Figure 7 An example implementation is shown in which two distributed failure resilient address spaces reside on multiple solid-state storage disks. Block 710 1,1 to 710 D,C Can be organized into multiple block strips 7201-720 S (S is an integer). In an example implementation, forward error correction (eg, erasure coding) is used to protect each chunk stripe 720 separately. s (s is an integer where 1≤s≤S). Thus, any particular chunk stripe 720 may be determined based on the desired level of data protection. s Block 710 d,c The number of

[0102] For the purpose of illustration, assume that each block stripe 720 s The number of blocks 710 is N=M+K (where each of N, M and K is an integer). d,c , then N blocks 710 d,c The M of them can store data codes (usually binary digits or "bits" for current storage devices) and the N blocks 710 d,c The K of them can store protection codes (again usually bits). The virtual file system can then send each stripe 720 s Allocate N chunks from N different fault domains 710 d,c .

[0103] As used herein, a "fault domain" refers to a group of components where a failure in any one component (a component loses power, becomes unresponsive, etc.) may result in the failure of all components. For example, if a rack has a single top-of-rack switch, a failure of that switch will disconnect all components (e.g., compute, storage, and / or VFS nodes) on that rack. Thus, to the rest of the system, this is equivalent to all components on that rack failing together. A virtual file system according to the present disclosure may include fewer fault domains than chunk 710.

[0104] In an example implementation where the nodes of the virtual file system are connected and powered with only a single storage device 706 in a fully redundant manner for each such node, the failure domain may be only that single storage device 706. Thus, in an example implementation, each chunk stripe 720 s Includes storage devices 7061-706D A plurality of blocks 710 on each of N d,c (Thus, D is greater than or equal to N). An example of such an implementation is Figure 7 Displayed in.

[0105] exist Figure 7 In FIG. 7 , D=7, N=5, M=4, K=1, and the storage devices are organized into two DFRAS. These numbers are for illustration only and are not intended to be limiting. The three chunk stripes 720 of the first DFRAS are arbitrarily called out for illustration. The first chunk stripe 7201 consists of chunks 710 1,1 ,710 2,2 ,710 3,3 ,710 4,5 and 710 5,6 The second block stripe 7202 is composed of block 710 3,2 ,710 4,3 ,710 5,3 ,710 6,2 and 710 7,3 The third block stripe 7203 is composed of block 710 1,4 ,710 2,4 ,710 3,5 ,710 5,7 and 710 75 composition.

[0106] Although in actual implementation Figure 7 In the simple example of , D=7 and N=5, but D may be much larger than N (e.g., greater than an integer multiple of 1, and possibly up to multiple orders of magnitude), and both values ​​may be selected so that the probability that any two chunk stripes 720 of a single DFRAS reside on the same set of N storage devices 706 (or more generally, on the same set of N failure domains) is below a desired threshold. In this way, any single storage device 706 d The failure of any particular stripe 720 (or more generally, any single failure domain) will result (expected statistics may be determined based on the selected values ​​of D and N, the size of the N storage devices 706, and the arrangement of the failure domains) in s At most one block 710 b,c Furthermore, a double failure will cause the majority of stripes to lose at most a single block 710 b,c , and only a small number of stripes (determined based on the values ​​of D and N) will be removed from any particular strip 720 s two chunks are lost in a single failure (e.g., the number of stripes with two failures may be exponentially smaller than the number of stripes with one failure).

[0107] For example, if each storage device 706 dis 1 TB, and each block is 128 MB, then storage device 706 d The failure of will result in (expected statistics may be determined based on the following: the selected values ​​of D and N, the size of the N storage devices 706, and the arrangement of the failure domains) 7812 (=1TB / 128MB) chunk stripes 720 losing one chunk 710. For each such affected chunk stripe 720 s , an appropriate forward error correction algorithm and block stripe 720 may be used s The other N-1 chunks are used to quickly rebuild the lost chunk 710 d,c In addition, since the affected 7812 chunk stripes are evenly distributed across all storage devices 7061-706 D Therefore, the lost 7812 chunks 710 are reconstructed d,c will involve (the desired statistics may be determined based on the selected values ​​of D and N, the size of the N storage devices 706, and the arrangement of the failure domains) D Reading the same amount of data (ie, the burden of rebuilding the lost data) is evenly distributed across all storage devices 7061-706 D to recover very quickly from failures).

[0108] Next, turn to the two storage devices 7061–706 D In the case of concurrent failures (or more generally, concurrent failures of two fault domains), since each DFRAS block stripe 7201-720 S In all storage devices 7061-706 D Evenly distributed on the S Two of the N chunks are lost. The virtual file system is operable to quickly identify such a twice lost chunk stripe based on metadata indicating that the chunk stripes 7201-720 S With storage device 7061-706 D Once such twice lost chunk stripes are identified, the virtual file system can prioritize rebuilding those twice lost chunk stripes before starting to rebuild the once lost chunk stripes. The remaining chunk stripes will have only a single lost chunk, and for them (the vast majority of the affected chunk stripes), the two storage devices 706 d Concurrent failure with only one storage device 706 d A similar principle applies to three concurrent failures (in a two concurrent failure scenario, the number of chunk stripes with three failed blocks will be much less than the number with two failed blocks), and so on. In an example implementation, chunk stripes 720 may be used based on the chunk stripe 720.s The number of losses in the control execution block stripe 720 s This can be achieved by, for example, controlling the rate at which reads and commits for reconstruction are performed, the rate at which FEC calculations for reconstruction are performed, the rate at which network messages for reconstruction are transmitted, etc.

[0109] Figure 8 A forward error correction scheme according to an example implementation of the present disclosure is shown, which can be used to protect data stored in a non-volatile memory of a virtual file system. A storage block 902 of a block stripe 7301-7304 of a DFRAS is shown 1,1 –902 7,7 .exist Figure 8 In the protection scheme of , five blocks of each stripe are used to store data codes, and two blocks of each stripe are used to store data of protection codes (i.e., M=5 and K=2). Figure 8 The protection code is calculated using the following formulas (1)-(9):

[0110]

[0111] therefore, Figure 8 The four stripes 7301-7304 in are part of a multi-strip (in this case, four stripes) FEC protection domain, and the loss of any two or fewer blocks in any block stripe 7301-7304 is recovered by using various combinations of equations (1)-(9) above. For comparison, an example of a single-strip protection domain is: if D11, D22, D33, D44, D54 are protected only by P1, and D11, D22, D33, D44, D54 and P1 are all written to stripe 7301 (7301 is a single-strip FEC protection domain).

[0112] According to an example implementation of the present disclosure, a plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices includes one or more of a plurality of storage devices. A plurality of failure resilient address spaces are distributed over the plurality of storage devices, such that each of the plurality of failure resilient address spaces spans across the plurality of storage devices. Each of the plurality of failure resilient address spaces is organized into a plurality of stripes (e.g., Figure 7 and Figure 8 Each or more of the plurality of stripes is a plurality of forward error correction (FEC) protection domains (e.g., such as Figure 8 Each of the plurality of stripes may include a plurality of storage blocks (e.g., a plurality of 512). Each block of a particular stripe in the plurality of stripes may be located on a different storage device in the plurality of storage devices. A first portion of the plurality of storage blocks (e.g., Figure 8Strip 7301 of 902 1,2 –902 1,6 The number of 5 composed of the plurality of storage blocks) can be used to store the data code, and the second part of the plurality of storage blocks (for example, Figure 8 The strip 7301 of two 902 1,1 and 902 1,7 The number of) can be used to store protection codes calculated at least in part based on data codes. Multiple computing devices can be operable to sort multiple stripes. The sorting can be used to select which of the multiple stripes is used for the next commit operation of one of the multiple fault-resilient address spaces. The sorting can be based on how many protected and / or unprotected storage blocks there are in each of the multiple stripes. For any of the multiple stripes, the sorting can be based on a bitmap stored on multiple storage devices having a specific one of the multiple stripes. The sorting can be based on the number of blocks currently storing data in each of the multiple stripes. The sorting can be based on the read and write overhead for committing to each of the multiple stripes. Each fault-resilient address space can be owned by only one of the multiple computing devices at any given time, and each of the multiple fault-resilient address spaces can only be read and written by its owner. Each computing device can own multiple fault-resilient address spaces. Multiple storage devices can be organized into multiple fault domains. Each of the multiple stripes can span multiple fault domains. Each failure resilient address space may span all of the multiple failure domains such that when any particular failure domain in the multiple failure domains fails, the workload for rebuilding the lost data is distributed between each other failure domain in the multiple failure domains. The plurality of stripes may be distributed across the multiple failure domains such that in the event of a simultaneous failure of two of the multiple failure domains, the probability of two blocks in any particular stripe in the multiple stripes on the two failure domains in the multiple failure domains being present is less than the probability of only one block in any particular stripe in the multiple stripes on the two failure domains in the multiple failure domains being present. The plurality of computing devices may be used to first rebuild any of the multiple stripes with two failed blocks and then rebuild any of the multiple stripes with only one failed block. The plurality of computing devices may be used to perform the reconstruction of the multiple stripes with two failed blocks at a higher rate (e.g., a greater percentage of CPU clock cycles dedicated to reconstruction, a greater percentage of network transmission opportunities dedicated to reconstruction, etc.) than the reconstruction rate of the multiple stripes with only one failed block. The plurality of computing devices may be used to determine the rate at which any particular lost block is rebuilt based on how many other blocks of the same stripe in the multiple stripes are lost in the event of a failure of one or more failure domains. Wherein, one or more of the plurality of fault domains include a plurality of storage devices. Each of the plurality of FEC protection domains may span a plurality of stripes of the plurality of stripes. The plurality of stripes may be organized into a plurality of groups (e.g., Figure 7 Chunk strips 7201-720 inS ), wherein each of the plurality of groups includes one or more of the plurality of stripes, and the plurality of computing devices are operable to sort, for each group, one or more of the plurality of stripes of the group. The plurality of computing devices are operable to: perform continuous commit operations on a selected one of the plurality of groups until one or more of the plurality of stripes of the group no longer satisfies a determined criterion, and select another one of the plurality of groups when the selected one of the plurality of groups no longer satisfies the determined criterion. The criterion may be based on how many blocks are available for new data to be written.

[0113] Although the present method and / or system has been described with reference to certain implementations, it will be appreciated by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present method and / or system. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the present invention without departing from the scope of the invention. Therefore, it is intended that the present method and / or system is not limited to the particular implementation disclosed, but rather that the present method and / or system will include all implementations that fall within the scope of the appended claims.

[0114] As used herein, the terms "circuit" and "circuitry" refer to physical electronic components (i.e., hardware) and any software and / or firmware ("code") that can configure hardware, be executed by hardware, and / or be associated with hardware. As used herein, for example, a specific processor and memory may include a first "circuit" when executing a first line or more lines of code, and may include a second "circuit" when executing a second line or more lines of code. As used herein, "and / or" refers to any one or more items connected by "and / or" in a list. As an example, "x and / or y" represents any element in a three-element set {(x), (y), (x, y)}. In other words, "x and / or y" refers to "one or both of x and y". As another example, "x, y and / or z" represents a seven-element set {(x), (y), (z), (x, y), (x, z), (y, z), (x, y, z)}. In other words, "x, y and / or z" refers to "one or more of x, y and z". As used herein, the term "exemplary" means serving as a non-limiting example, instance, or illustration. As used herein, the terms "for example" and "for example" lead to a list of one or more non-limiting examples, instances, or illustrations. As utilized herein, a circuit is "operable" to perform a function so long as the circuit contains the necessary hardware and code (if necessary) required to perform that function, regardless of whether the performance of that function is disabled (e.g., by a user-configurable setting, a factory adjustment, etc.).

Claims

1. A method for recovering when a first server is inaccessible, include: Rebuilding the data block of the first server; as well as The reconstructed data blocks are stored in the second server, wherein: The data blocks in the first server are associated with a first plurality of stripes, a second plurality of stripes including said reconstructed data blocks, Each stripe in the first plurality of stripes and the second plurality of stripes spans a different set of servers, and Each different set of servers includes at least two servers.

2. The method according to claim 1, in, If write access to the first server fails, it is determined that the first server is inaccessible.

3. The method according to claim 1, in, The method includes performing a degraded mode read by the following steps: Rebuilding the first data block in the first server using stripe data blocks associated with the first data block, wherein each of the stripe data blocks is located in a unique server different from the first server; and The reconstructed first data block is populated in the read cache.

4. The method according to claim 1, in, The method includes redirecting a write request to another server associated with the data block for which the write request is intended.

5. The method according to claim 1, in: performing a degraded mode read for each read request directed to the first server before the wait time has elapsed; and Before the waiting time has elapsed, each write request directed to the first server is redirected to another server.

6. The method according to claim 1, in, Rebuilding the data blocks of the first server includes: for each of the data blocks, using a stripe data block associated with each of the data blocks of the first server.

7. The method of claim 6, comprising reading all stripe data blocks in parallel.

8. The method according to claim 1, include: When the first server is one of the inaccessible servers, data blocks to be reconstructed in the first server are prioritized.

9. The method according to claim 8, in, The prioritization includes rebuilding data chunks based on how many servers are inaccessible.

10. The method according to claim 1, include: receiving a notification from the first server that the first server is online; determining whether the data in the first server is not damaged; as well as When it is determined that the data is not corrupted, any data blocks that were written to another server while the first server was inaccessible are moved to the first server.

11. A system comprising a plurality of storage devices controlled by a plurality of servers, the system include: One or more processors configured to: Determining that the first server is inaccessible, rebuilding the data block of the first server, determining a second server to replace the first server, and The reconstructed data block is stored in the second server, wherein: The data blocks in the first server are associated with a first plurality of stripes, a second plurality of stripes including said reconstructed data blocks, Each stripe in the first plurality of stripes and the second plurality of stripes spans a different set of servers, and Each different set of servers includes at least two servers.

12. The system according to claim 11, in, The one or more processors are configured to perform a read in a degraded mode: Rebuilding the first data block in the first server using stripe data blocks associated with the first data block, wherein each of the stripe data blocks is located in a unique server different from the first server; and The reconstructed first data block is populated in the read cache.

13. The system according to claim 11, in, The one or more processors are configured to redirect a write request to another server associated with the data block for which the write request is intended.

14. The system according to claim 11, in, When the first server is one of the inaccessible servers, the one or more processors are configured to prioritize rebuilding the data blocks on the first server by determining one or more data blocks most affected by the inaccessible servers.

15. The system according to claim 11, in, The one or more processors are configured to: receiving a notification from the first server that the first server is online; determining whether the data in the first server is not corrupted; and When it is determined that the data is not corrupted, any data blocks that were written to another server while the first server was inaccessible are moved to the first server.