Meta Replica Set for Fault-Tolerant Data Storage

Abstract

A system includes determining multiple meta-replica sets of multiple storage devices, each meta-replica set in the meta-replica set includes a first number of storage devices; determining a first replica of a second number of storage devices from a first meta-replica set in the meta-replica set, the second number being less than the first number; using the first replica set to store first data in a fault-tolerant format; determining to migrate the first data from the first storage device of the first replica set; and in response to determining to migrate the first data, determining a second replica set of a second number of storage devices, the second replica set including storage devices from the first meta-replica set that are not in the first replica set and storage devices of the first replica set other than the first storage device of the first replica set, and using the second replica set to store the first data in a fault-tolerant format.

Description

Technical Field

[0001] The present disclosure relates generally to the field of data storage and, in particular, to devices, systems, and methods for storing data in a fault-tolerant format across multiple storage devices. Background Art

[0002] A replica set is a collection of machines that store copies of the same data block. As long as at least one machine in the replica set remains available, at least one copy of the data block will always be available. Current storage systems assign each stored data block to a random replica set. Therefore, if a certain number of machines become unavailable simultaneously, it is possible that all machines in at least one replica set will become unavailable, and therefore the data block stored in that replica set will become unavailable.

[0003] It has been observed that limiting the number of replica sets in use can reduce the probability that all machines in any replica set will become unavailable at any given time. Therefore, in some systems, each data block is stored in one of a limited number of replica sets, rather than in a random replica set of machines. Because a limited number of replica sets are used, if all machines in a replica set become unavailable, many data blocks will become unavailable. This result is considered acceptable compared to the above system because the reduction in data unavailability in the event of a replica set failure is generally more significant than the negligible increase in recovery costs.

[0004] When using a limited replica set, migrating data from a single machine in the replica set can be expensive. Such a migration might be necessary if the machine is unavailable or nearing its storage capacity. However, the option to migrate data from a single machine to another machine in the same replica set is typically not provided.

[0005] Storage using a limited replica set also assumes that data chunks are copied across replica set members. Some storage systems implement erasure coding to use storage more efficiently. Data can become unavailable in such systems if several machines become unavailable simultaneously. Specifically, if all copies of a chunk are stored on a failed machine, the chunk becomes unavailable. If a chunk is erasure-coded with X+Y fragments and the machine storing Y+1 fragments becomes unavailable, the chunk will also become unavailable.

[0006] Expected to address the shortcomings of limited replica sets and support systems that store erasure-coded data. Summary of the Invention

[0007] In a first aspect, a device for data storage is provided, comprising: a memory storing processor-executable process steps; one or more processors for executing the processor-executable process steps so that the device: determines multiple meta-replica sets of multiple storage devices, each meta-replica set in the meta-replica set includes a first set of a first number of storage devices; determines a first replica set including a second number of storage devices from the first meta-replica set in the meta-replica set, wherein the second number is less than the first number; uses the first replica set to store first data in a fault-tolerant format; determines to migrate the first data from the first storage device of the first replica set; and in response to determining to migrate the first data: determines a second replica set of the second number of storage devices, the second replica set including storage devices from the first meta-replica set that are not in the first replica set, and storage devices of the first replica set other than the first storage device of the first replica set, and uses the second replica set to store the first data in a fault-tolerant format.

[0008] In a second aspect, a system for data storage is provided, comprising: a plurality of storage devices; and a server for storing data in the plurality of storage devices, the server being configured to: determine a plurality of first sets of storage devices, each first set in the first sets including a first number of storage devices; receive first data to be stored; determine a first subset of a second number of storage devices from a first first set in the first sets of storage devices, wherein the second number is smaller than the first number; store the first data in a fault-tolerant format using the first subset of the second number of storage devices; determine a first storage device to replace the first subset of the second number of storage devices; and in response to determining to replace the first storage device: determine a replacement storage device from the first first set in the first set of storage devices, wherein the first subset of the second number of storage devices does not include the replacement storage device; and store the first data in a fault-tolerant format using the replacement storage device and the first subset of the second number of storage devices that does not include the first storage device.

[0009] In a third aspect, a method for storing data in a fault-tolerant format between multiple storage devices is provided, the method comprising: determining multiple first sets of storage devices, each first set in the first sets including a first number of storage devices; receiving first data to be stored; determining a first subset of a second number of storage devices from a first first set in the first sets of storage devices, wherein the second number is less than the first number; using the first subset of the second number of storage devices to store the first data in a fault-tolerant format; determining a first storage device to replace the first subset of the second number of storage devices; and in response to determining to replace the first storage device: determining a replacement storage device from the first first set in the first set of storage devices, wherein the first subset of the second number of storage devices does not include the replacement storage device; and using the replacement storage device and the first subset of the second number of storage devices that does not include the first storage device to store the first data in a fault-tolerant format. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A storage architecture according to some embodiments is illustrated.

[0011] Figure 2 According to some embodiments Figure 1 A tabular representation of the schema's meta-replica set.

[0012] Figure 3 is a tabular representation of a mapping of data to replica sets according to some embodiments.

[0013] Figure 4 Illustrated is a storage architecture including an unavailable machine according to some embodiments.

[0014] Figure 5 According to some embodiments Figure 4 A tabular representation of the schema's meta-replica set.

[0015] Figure 6 is a tabular representation of a mapping of data to revisions of replica sets according to some embodiments.

[0016] Figure 7 is a flow diagram of a process for storing data in a replica set of a meta replica set, according to some embodiments.

[0017] Figure 8 Illustrated is a storage architecture including unavailable failure domains according to some embodiments.

[0018] Figure 9 According to some embodiments Figure 8 A tabular representation of the schema's meta-replica set.

[0019] Figure 10 is a tabular representation of a revision mapping of data to replica sets according to some embodiments.

[0020] Figure 11 Illustrated is the storage of erasure coded segments in a group of machine racks in accordance with some embodiments.

[0021] Figure 12 Illustrated is the migration of erasure coded segments across a set of machine racks in accordance with some embodiments.

[0022] Figure 13 Illustrated is the migration of erasure coded segments across a set of machine racks in accordance with some embodiments.

[0023] Figure 14 Illustrated is a computing device for facilitating data storage in accordance with some embodiments. DETAILED DESCRIPTION

[0024] The following description is provided to enable any person skilled in the art to make and use the described embodiments. However, various modifications will still be apparent to those skilled in the art.

[0025] Some embodiments solve the aforementioned problems by using meta-replica sets. A meta-replica set is a collection of machines from which one or more replica sets are generated. For example, in a system implementation that provides three-way data copying, a replica set of three machines (i.e., of size 3) can be defined from a meta-replica set of size 5. As described above, a particular data block can only be stored in the defined machine replica sets. Assuming two meta-replica sets ABCDE and ABFGH, examples of valid replica sets that can be used to store data include ABC, ABD, and BGH, but not CDH.

[0026] Therefore, data can generally be migrated from one machine to another while adhering to a limited set of replica sets. Furthermore, while using meta-replica sets significantly increases the number of replica sets eligible for storage, the risk of data unavailability is not significantly increased due to the reuse of machines (and the resulting higher failure correlation) across replica sets in the same meta-replica set.

[0027] Data migration may be desirable for a variety of reasons, the primary of which are failure recovery and storage load balancing. In the first case, a machine is expected to fail long-term and its data needs to be rebuilt on other machines. In the second case, a machine is identified as having reached its storage capacity, while one or more other machines are identified as having more free storage capacity, and therefore it is desirable to move data from the previous machine to the next machine(s).

[0028] Unlike traditional replica set-based systems, some embodiments provide efficient data migration for each machine in the replica set, so that all migrated data blocks remain on the valid replica set. Embodiments can be used for copy-based fault tolerance and erasure coding storage implementations, which will be described in more detail below.

[0029] Figure 1 A storage architecture 100 is illustrated, according to some embodiments. The storage architecture 100 can store any type of data from any source. The storage architecture 100 can implement an on-premises data storage system, a cloud-based storage service, a hybrid storage system, and / or any other type of data storage system in any deployment that is or becomes known. According to some embodiments, the storage architecture 100 is designed to store data in a fault-tolerant manner.

[0030] Storage architecture 100 includes a storage engine 110. Storage engine 110 will be described herein as controlling the receipt and storage of incoming data, and providing data in response to requests. According to some embodiments, storage engine 110 operates in conjunction with metadata service 120. Metadata service 120 can define meta-replica sets, determine replica sets from meta-replica sets and assign data chunks to them, and determine and reassign replica sets to data chunks to assist in data migration.

[0031] According to some embodiments, a data block stored in a replica set may include any identifiable unit of data. For example, a data block may include an extent, which is a collection of eight physically contiguous 8KB pages. In some embodiments, a page begins with a 96-byte header that stores the page number, page type, the amount of free space on the page, and the allocation unit ID of the object that owns the page. In X-way copy-based storage, each replica set consists of X machines, and an extent is replicated in its entirety to each of the X machines in its assigned replica set.

[0032] According to some embodiments, a data block may include fragments generated by erasure-based coding. According to erasure-based coding, the data to be stored (e.g., an extent) is divided into X (e.g., 6) data fragments of equal size. Coding is applied to the data fragments to generate Y (e.g., 3) parity fragments, each of which is the same size as the data fragment. With the help of the coding scheme, the original data can be reconstructed from any X (i.e., 6) of these X+Y (i.e., 9) fragments. Fault tolerance is achieved by storing the X+Y fragments across X+Y different machines, so that the data can be recovered even if Y machines fail. In addition, the storage space required by this storage arrangement is equal to 1.5 times the size of the original data, while in the case of a copy, it is 3 times.

[0033] Architecture 100 includes six fault domains 130 through 135. Each fault domain includes several machines. All machines within a fault domain are susceptible to simultaneous unavailability. An example of a fault domain includes a network rack, as each machine installed in a network rack is typically connected to the same power supply and the same top-of-rack switch. Embodiments are not limited to machines sharing a corresponding fault domain. Some embodiments include a group of machines in which one, some, or all of the machines do not share a fault domain with any other machine.

[0034] A machine as described herein may include any computing device or system in which data may be stored. According to some embodiments, the machine is a blade server, and each failure domain is a network rack in which the blade server is installed.

[0035] Each element depicted in architecture 100 may include any number of computing devices and any type of hardware or software for performing the functions attributed thereto. Each element may be located adjacent to or remote from any one or more other elements, and the elements may communicate with each other using any number and type of intervening computing networks. Storage architectures according to some embodiments may include Figure 1 Components not depicted.

[0036] Figure 2 This is a tabular representation of metadata 200 for meta-replica sets that have been defined based on the machines of architecture 100. According to some embodiments, meta-replica sets are determined by metadata service 120 and metadata 200 is maintained within storage architecture 100. Meta-replica sets are determined based on one or more constraints. One constraint is the number of machines included in each meta-replica set.

[0037] The desired number of machines in the meta-replica set is based on the number of machines C in the replica set. The number of machines C, in turn, is based on the type of fault-tolerant data storage scheme to be used. In the case of three-way replication, C = 3. In the case of 14+4 erasure coding, C = 18. According to some embodiments, the number of machines in the meta-replica set is C + 2. Embodiments are not limited thereto, and C + 1, C + 3, C + 4, etc. machines may be employed in each replica set.

[0038] In some embodiments, the determination of the meta-replica set is also subject to rules associated with the properties of the machines in the meta-replica set. For example, the rules may require that each machine in the meta-replica set be associated with a different failure domain.

[0039] Figure 2 Four meta-replica sets 202, 204, 206, and 208 are depicted, but the architecture 100 does not limit the number of meta-replica sets to this. As shown, each meta-replica set includes 5 machines (assuming C=3), and each machine in the meta-replica set belongs to a different failure domain. In addition, no machine belongs to more than one meta-replica set.

[0040] Figure 3 is a tabular representation of metadata 300 that associates data chunks with replica sets according to some embodiments. Metadata service 120 can assign each data chunk to be stored to a replica set, which in turn is determined by one of the meta-replica sets defined in metadata 200. As shown, each replica set identified by metadata 300 includes machines from a single meta-replica set.

[0041] Metadata 300 includes replica sets that, although not identical, are subsets of the same meta-replica set. For example, the replica sets for block 0 and block 3 are each associated with meta-replica set 208, and the replica sets for block 4 and block 6 are each associated with meta-replica set 202. Thus, data blocks associated with these different replica sets are associated with the same meta-replica set. Metadata 300 also assigns some data blocks (e.g., (block 2, block 5) and (block 0, block 7)) to the same replica set (e.g., (C3, D4, E4) and (D3, E3, F2)).

[0042] In general, N meta-replica sets yield N(C+1)(C+2) / 2 possible replica sets because there are (C+1)(C+2) / 2 ways to choose a replica set of size C from a meta-replica set of size C+2. This increase in replica sets results in only a modest increase in the likelihood of data unavailability compared to traditional replica set implementations. This increase is modest because the replica sets of a meta-replica set include general-purpose machines, so an event that causes one replica set to become unavailable is likely to cause other replica sets of the meta-replica set to become unavailable. Furthermore, using meta-replica sets as described herein can provide a significant benefit: the ability to perform failure recovery and load balancing without having to migrate large amounts of data to entirely new replica sets.

[0043] According to some embodiments, any replica set that is a subset of the meta replica set is a valid replica set. As a result, data can be efficiently migrated from a machine without violating replica set restrictions. Figures 4 to 6 An example of data migration according to some embodiments is illustrated.

[0044] like Figure 4 As shown, assume that machine C3 of failure domain 132 has become unavailable. In some embodiments, this unavailability is detected by metadata service 120. In response, metadata service 120 determines to migrate data chunks residing on the replica set that includes machine C3 to another machine. As described herein, this other machine and machine C3 belong to the same meta-replica set. Because a meta-replica set can include many machines, the machine to which a data chunk stored on machine C3 is migrated may be different from the machine to which another data chunk stored on machine C3 is migrated.

[0045] Return Reference Figure 3 , chunk 2 and chunk 5 are each assigned to a replica set that includes machine C3. Therefore, the replicas of chunk 2 and chunk 5 reside on machine C3. Due to the unavailability of machine C3, chunk 2 and chunk 5 must be replicated to another machine to maintain a three-way copy in this example.

[0046] As shown in metadata 300, chunks 2 and 5 are each associated with a replica set that includes machines C3, D4, and E4. Embodiments are not limited in this regard, as each chunk can be associated with a different replica set, each of which includes machine C3. According to some embodiments, the machines to which chunks 2 and 5 are replicated in response to the unavailability of machine C3 and the replica sets associated with chunks 2 and 5 in metadata 300 belong to the same meta-replica set.

[0047] Replica sets C3, D4, and E4 belong to meta-replica set 204. Therefore, an available machine is selected from meta-replica set 204 to replace machine C3 in the replica set associated with chunks 2 and 5. Figure 6 The metadata 300 is shown modified to reflect the replacement. Specifically, machine A3 has replaced machine C3 associated with chunk 2 in the replica set, and machine B1 has replaced machine C3 associated with chunk 5 in the replica set. Figure 5 As shown, replacement machines A3 and B1 both belong to meta-replica set 204, thereby preserving the meta-replica set / replica set restrictions described above.

[0048] Figure 7 A flowchart of a process 700 for providing fault-tolerant data storage according to some embodiments is included. In some embodiments, a processing unit (e.g., one or more processors, processing cores, processor threads) of a data storage server executes software program code to cause the server to perform process 700. The execution of process 700 can be distributed between multiple data storage server nodes. Process 700 and all other processes mentioned herein can be embodied in a processor executable program code, which is read from one or more of a non-transitory computer-readable medium (e.g., a floppy disk, CD-ROM, DVD-ROM, flash drive, and tape) and then stored in a compressed, uncompiled, and / or encrypted format. In some embodiments, a hard-wired circuit can be used to replace program code or be combined with program code to implement a process according to some embodiments. Embodiments are therefore not limited to any particular combination of hardware and software.

[0049] Initially, at S710, a plurality of meta-replica sets are determined. Each meta-replica set includes a number of storage devices (referred to as machines above). As described above, at S710, the meta-replica sets are determined based on one or more constraints. One constraint is the number of storage devices to be included in each meta-replica set. The number of storage devices in each determined meta-replica set is greater than the number of storage devices required for the specific fault-tolerant storage scheme used. Another constraint may be the relative position of the storage devices within the meta-replica set. According to one constraint, only a limited number (e.g., 0, 2) of storage devices in the meta-replica set can belong to the same fault domain.

[0050] The process for determining multiple meta-replica sets will now be described. The process assumes that each replica set should include C storage devices, and each meta-replica set should include C+2 storage devices. Furthermore, each storage device is associated with a known failure domain (e.g., a rack), where the known failure domain is shared by one or more other storage devices.

[0051] The process begins by randomly permuting the racks. Next, the storage devices within each rack are randomly permuted. The permutations are merged into lists by sorting the storage devices using the primary sort key, "Device position within permutation of devices within the same rack," and the secondary sort key, "Device rack position within permutation of racks." The resulting lists are partitioned into groups, where each group is a meta-replica set consisting of C + 2 machines. The above steps are repeated until the desired dispersion width is achieved.

[0052] In this regard, the spread width of storage device M is the number of other devices in the same replica set as M. Since the spread width indicates the number of storage devices that may be available to help recover or move M's data, a higher spread width may be desirable. In the case of a meta replica set, an increased spread width increases the number of devices that can be selected when moving data off a machine.

[0053] The primary way to increase dispersion width is to use more meta-replica sets. The more meta-replica sets that M is part of, the more other devices will share replica sets with M. However, using too many meta-replica sets is problematic because the increased number of meta-replica sets increases the probability of data unavailability. A goal is to use enough meta-replica sets to provide a reasonable dispersion width for each device, while assigning devices to meta-replica sets to provide the maximum dispersion width possible for a small number of meta-replica sets. To this end, it is desirable to have little overlap between meta-replica sets.

[0054] A reasonable target for the spread width S is Z / 2, where Z is the number of devices on other racks. For example, in an architecture with 20 racks and 16 devices per rack, Z is 19*16. Accordingly, for a cluster with which a device could potentially share a meta-replica set, the device would share a meta-replica set with half of all devices in the cluster.

[0055] The repeat count R is to repeat the four steps to achieve the desired dispersion width times. From the perspective of a single device, each time the four steps above are repeated, C+1 devices are added to the set of devices that share the meta-replica set. However, some of these devices are already in the set, so the scatter width does not necessarily increase by a full C+1. This process is essentially sampling from Z devices with replacement. When selecting m items from n items with replacement, the formula for the expected number of different items is Therefore, the appropriate formula is In the constraints In the case of For example, if C=18 and Z=19*16, then R=11.

[0056] Once the devices for the multiple meta-replica sets are determined, the process proceeds to S720. Next, the data to be stored is assigned to the corresponding replica sets within the meta-replica set. For example, for the first data block to be stored, a subset of the storage devices of the first meta-replica set in the meta-replica set is determined. The size of the subset is smaller than the size of the meta-replica set and is determined by the type of fault-tolerant data storage scheme to be used. If three-way copy is to be used, the subset includes three storage devices. If 6+3 erasure coding is to be used, the subset includes nine storage devices.

[0057] At S730, the first data is redundantly stored on a subset of the storage devices determined for the first data at S720. In a copy-based storage system, a complete copy of the first data is stored on each of the three determined storage devices. If the storage system uses erasure coding, the first data is divided into fragments, and a number of additional fragments are generated from the original fragment, and each fragment is stored on a corresponding one of the determined storage devices.

[0058] S720 and S730 may be repeated for any number of data elements to be stored, thereby generating metadata associating each data element with a replica set, e.g. Figure 3 and Figure 6 300 of metadata.

[0059] S740 to S760 can be as described above. Figures 4 to 6Specifically, metadata service 120 may detect the unavailability of a storage device in the subset of storage devices of the first meta-replica set. In response, at S750, metadata service 120 determines a storage device in the first meta-replica set that is not in the subset of storage devices determined for the first data. The first data is then copied to the storage device determined at S750.

[0060] S740 to S760 may be repeated for other data stored on various subsets of storage devices including unavailable storage devices. The storage devices to which other data is migrated may be as described above. Figure 6 The storage devices described are different.

[0061] Figure 8 The diagram shows a situation where all machines in the failure domain are unavailable. Once the unavailability is detected, a replacement machine is determined for the replica set that includes one of the unavailable machines. Figure 10 shown and with Figure 6 Compared to the replica sets of , each replica set that includes one of machines E1, E2, E3, or E4 has been updated to replace the unavailable machine with another available machine from the same meta-replica set.

[0062] Erasure coding schemes may require more machines than racks in a cluster, for example, in the case of 20+4+2 coding in a cluster with only 20 racks. Therefore, a meta-replica set can include machines that are located on the same rack. In such a scheme, metadata can specify certain fragment pairs that are allowed to occupy machines in the same rack. However, it is desirable to minimize the amount of rack reuse within the meta-replica set. The above meta-replica set determination can minimize the number of machines that share racks. For example, if the meta-replica set size is 28, the determination will first allocate 20 machines on 20 different racks to the meta-replica set, and then reuse 8 racks for the remaining 8 machines of the replica set.

[0063] In order to maintain the constraint that only certain pairs of fragments can be stored on machines in the same rack, in some cases moving a fragment due to machine unavailability will require moving one or more additional fragments. For example, Figure 11 The storage of data comprising 26 segments is illustrated. Figure 11 There are six machines called M1, M2, M3, M4, M5, and M6, and 22 other machines. M1 and M2 are on the same rack, and M3, M4, M5, and M6 are on different racks. The 26 fragments are stored on all 28 machines except M5 and M6.

[0064] It will be assumed that fragments "x" and "y" can be located on machines in the same rack, but two fragments "x" or two fragments "y" cannot be located on machines in the same rack. If both M1 and M2 fail simultaneously, fragment x0 on M1 and fragment y0 on M2 must be moved to other clusters in the meta replica set. Figure 12 The completed migration of fragment x0 and fragment y0 to two standby machines M5 and M6 of the meta replica set is shown.

[0065] Now assume that M1 and M2 eventually come back online, and then M3 and M4 fail. Fragment x1 must be recreated from M3, and x2 must be recreated from M4, and M1 and M2 are available but currently unused. However, storing x1 and x2 on M1 and M2 is not allowed, because x1 and x2 should not be on the same rack. This can be done as Figure 13 As shown, this is solved by first moving x0 from M5 to M1 and y0 from M6 to M2, thereby making room for x1 and x2 on M5 and M6.

[0066] Certain events (e.g., a major heat intrusion) may cause many machines to be unavailable at the same time, making it impossible to store new data on the valid replica set. In addition, redundant segments of existing data may not be able to be moved to other machines in the associated meta replica set. In response to such a situation, data can be stored in a random replica set.

[0067] As mentioned above, the use of random replica sets can lead to high unavailability rates. Therefore, a background process can be employed to identify data that is not stored in the valid replica set (i.e., the replica set in which all machines belong to the defined meta-replica set) and migrate the data to the valid replica set. This background process can be allocated limited processing power and I / O bandwidth to prevent it from taking resources away from more important work.

[0068] In certain unavailability scenarios, the limited replica set solution proposed in this paper can only be satisfied if significant data movement is required. For example, a machine may fail in a long-term manner, necessitating the movement of its data to other machines. However, for a data block on that machine, the associated meta-replica set may include three unavailable machines. The limited replica set solution can be maintained by moving all data blocks to completely different meta-replica sets. However, this may require an undesirable amount of time and bandwidth. Therefore, when only one block needs to be migrated, the block is migrated even if the migration would violate the meta-replica set constraints. The background process described above can be used to ultimately resolve the above-mentioned violation.

[0069] Load balancing may be required to respond to machine failures and data recovery. Load balancing may also be required in situations where there are no failures or unrecoverable conditions. In this regard, random selection of replica sets will exhibit variability, and therefore place more data on some machines than on others. As some machines are closer to capacity than others, load balancing becomes desirable.

[0070] Because load balancing consumes bandwidth, it is preferably avoided. Therefore, the random selection of replica sets should be biased towards selecting replica sets with machines that have more available storage capacity. In some embodiments, two different replica sets are randomly selected as candidate replica sets for the new data block, one of the two replica sets is identified as having the largest amount of free space of any machine in the replica set, and the identified replica set is assigned to store the data block. Alternatively, the identified replica set is the replica set that minimizes the number of machines whose used storage capacity exceeds a certain threshold of the total capacity.

[0071] The above process incorporates the use of "new" machines that are empty because they previously failed and have been restored or introduced into the cluster as replacements for failed machines. However, depending on the rate at which data is generated, this introduction may not be fast enough to effectively use the new capacity. In this case, background load balancing can be used to increase the rate at which data is moved from fuller machines to empty ones. It may be desirable to limit the rate at which such background load balancing occurs to avoid using bandwidth, which is more preferably allocated to tasks such as servicing client requests.

[0072] New machines may present a highly correlated failure risk because all new machines are likely to come from a homogeneous pool and / or because new machines use hardware from crashed and refurbished machines, which are more likely to fail in the future. Therefore, placing data preferentially on new machines that are likely to crash at the same time may increase the risk of data unavailability. If this risk is determined to be unacceptable, sampling bias for load balancing purposes can be eliminated entirely, favoring background load balancing.

[0073] Figure 14 1 is a block diagram of a system 1400 according to some embodiments. System 1400 may include a general-purpose computer server and may execute program code to perform any of the processes described herein. Any one or more components of system 1400 may be implemented in a distributed architecture. According to some embodiments, system 1400 may include other elements not shown.

[0074] System 1400 includes a processing unit 1410 operatively coupled to a communication device 1420, a persistent data storage device 1430, one or more input devices 1440, one or more output devices 1450, and volatile memory 1460. Processing unit 1410 may include one or more processors, processing cores, etc. to execute program code. Communication device 1420 may be used to facilitate communication with external networks such as storage devices (i.e., machines) and client devices requiring data storage services. Input device(s) 1440 may include, for example, a keyboard, keypad, mouse or other pointing device, microphone, touch screen, and / or eye tracking device. Output device(s) 1450 may include, for example, a display (e.g., a screen), speakers, and / or a printer.

[0075] Data storage device 1430 may include any number of suitable persistent storage devices, including a combination of magnetic storage devices (e.g., magnetic tape, hard drive, and flash memory), optical storage devices, read-only memory (ROM) devices, etc. Memory 1460 may include random access memory (RAM), storage class memory (SCM), or any other fast access memory.

[0076] The storage engine 1431 and metadata service 1432 may include program code executed by the processing unit 1410 to cause the system 1400 to perform any one or more of the processes described herein. For example, the storage engine 1431 may control the storage of data in and the retrieval of data from multiple storage devices. The metadata service 1432 may determine the storage devices in which and / or to which data will be stored. These determinations may be based on the meta-replica set data 1433 and the data / replica set metadata 1434 described herein. The data storage device 1430 may also store data and other program code to provide additional functionality and / or functionality necessary for the operation of the system 1400 (e.g., device drivers, operating system files, etc.).

[0077] As is known in the art, each functional component described herein can be implemented in computer hardware (integrated circuits and / or discrete circuit components), program code, and / or in one or more computing systems that execute such program code. Such computing systems may include one or more processing units that execute processor-executable program code stored in a memory system.

[0078] The above-described diagram represents the logical architecture for describing process according to some embodiments, and actual implementation can include more or different components arranged in other ways.Other topologies can be used in combination with other embodiments.In addition, each assembly or device described herein can be realized by any number of devices communicating via any number of other public and / or private networks.Two or more such computing devices can be placed away from each other, and can communicate with each other via any known (multiple) network mode and / or dedicated connection.Each assembly or device can include any number of hardware and / or software elements suitable for providing function described herein and any other function.

[0079] The embodiments described herein are for illustrative purposes only. Those skilled in the art will appreciate that modifications and variations of the above embodiments may be made to practice other embodiments.

Claims

1. A device for data storage, comprising: a memory storing processor-executable process steps; one or more processors for executing the processor-executable process steps to cause the apparatus to: determining a plurality of meta-replica sets for a plurality of storage devices, each meta-replica set in the meta-replica sets comprising a first set of a first number of storage devices; determining a first replica set comprising a second number of storage devices from a first meta-replica set in the meta-replica sets, wherein the second number is less than the first number; storing the first data in a fault-tolerant format using the first replica set; Determine to migrate the first data from the first storage device of the first replica set; as well as In response to determining to migrate the first data: determining a second replica set of the second number of storage devices, the second replica set including storage devices from the first meta-replica set that are not in the first replica set, and storage devices of the first replica set other than the first storage device of the first replica set, and The first data is stored in a fault-tolerant format using the second replica set.

2. The apparatus of claim 1 , wherein storing the first data in a fault-tolerant format using the first replica set comprises copying the first data on each storage device in the first replica set, and Wherein storing the first data in a fault-tolerant format using the second replica set includes copying the first data on a storage device from the first meta-replica set that is not in the first replica set.

3. The apparatus of claim 1 , wherein storing the first data in a fault-tolerant format using the first replica set comprises: erasure encoding the first data to generate a second number of fragments associated with the first data, and storing each of the second number of fragments on a corresponding storage device of the first replica set, and Wherein using the second replica set to store the first data in a fault-tolerant format includes: regenerating fragments associated with the first data and stored on the first storage device, and storing the regenerated fragments on the storage device from the first meta-replica set that is not in the first replica set.

4. The apparatus of claim 3 , wherein each storage device in the plurality of storage devices is associated with a failure domain, the failure domain being shared with one or more other storage devices in the plurality of storage devices, wherein storing the regenerated fragment on a storage device from the first meta-replica set that is not in the first replica set comprises: identifying that the storage device from the first meta-replica set that is not in the first replica set is associated with a same failure domain as another storage device of the first replica set; as well as It is determined that the regenerated segment and the segment associated with the first data and stored on the other storage device of the first replica set are compatible with storage devices in the same failure domain.

5. The apparatus of claim 1 , wherein each storage device of the plurality of storage devices is associated with a failure domain, the failure domain being shared with one or more other storage devices of the plurality of storage devices, and wherein determining the plurality of meta-replica sets comprises: Randomly permuting the fault domains; randomizing the storage devices associated with each of the failure domains; merging the arrangement into a list, the list being ordered first by a position of each storage device in the arrangement of storage devices in the same failure domain, and second by a position of the storage device's failure domain in the arrangement of failure domains; as well as The first number of storage devices is divided into two or more groups, wherein each group is a meta-replica set of the plurality of meta-replica sets.

6. The apparatus of claim 1 , wherein the one or more processors are configured to perform the processor-executable process steps to cause the apparatus to: Determining to migrate second data from the first storage device of the first replica set; and In response to the determination to migrate the second data: determining that none of the replica sets of the second number of storage devices from the first meta-replica set are available; and In response to the determination that none of the replica sets of the second number of storage devices from the first meta-replica set are available, the second data is stored in a fault-tolerant format using a random replica set of the second number of storage devices among the plurality of storage devices.

7. The apparatus of claim 1 , wherein determining the first replica set of the second number of storage devices from the first meta-replica set in the meta-replica sets comprises: determining, from the first meta-replica set in the meta-replica sets, the first replica set of the second number of storage devices; determining a second candidate replica set for the second number of storage devices from a second meta-replica set in the meta-replica set; Determining the free storage capacity of the first replica set and the free storage capacity of the second candidate replica set; as well as The first replica set of the second number of storage devices is determined based on the free storage capacity of the first replica set and the free storage capacity of the second candidate replica set.

8. The apparatus of claim 1 , wherein the one or more processors are configured to perform the processor-executable process steps to cause the apparatus to: determining a third replica set from the second meta-replica set of the plurality of storage devices; storing third data in a fault-tolerant format using the third replica set; Determine to migrate the third data from the first storage device of the third replica set; and in response to the determination to migrate the third data: Determine a fourth replica set of the second number of storage devices, the fourth replica set including storage devices from the second meta-replica set that are not in the third replica set, and storage devices of the third replica set excluding the first storage device of the third replica set; as well as The first data is stored in a fault-tolerant format using the fourth replica set.

9. A system for data storage, comprising: Multiple storage devices; as well as A server is configured to store data in the plurality of storage devices, wherein the server is configured to: determining a plurality of first sets of storage devices, each of the first sets comprising a first number of storage devices; receiving first data to be stored; determining a first subset of a second number of storage devices from a first one of the first sets of storage devices, wherein the second number is less than the first number; storing the first data in a fault-tolerant format using the first subset of the second number of storage devices; determining a first storage device to replace said first subset of said second number of storage devices; as well as In response to determining to replace the first storage device: determining a replacement storage device from said first one of said first sets of storage devices, wherein said first subset of said second number of storage devices does not include said replacement storage device; as well as The first data is stored in a fault-tolerant format using the replacement storage device and the first subset of the second number of storage devices, excluding the first storage device.

10. The system of claim 9, wherein storing the first data in a fault-tolerant format using the first subset of the second number of storage devices comprises copying the first data on each storage device in the first subset of the second number of storage devices, and Wherein storing the first data in a fault-tolerant format using the replacement storage device and the first subset of the second number of storage devices excluding the first storage device includes copying the first data on the replacement storage device.

11. The system of claim 9 , wherein storing the first data in a fault-tolerant format using the first subset of the second number of storage devices comprises erasure encoding the first data to generate the second number of fragments associated with the first data, and storing each of the second number of fragments on a corresponding storage device in the first subset of the second number of storage devices, and Wherein storing the first data in a fault-tolerant format using the replacement storage device and the first subset of the second number of storage devices, excluding the first storage device, includes regenerating fragments associated with the first data and stored on the first storage device, and storing the regenerated fragments on the replacement storage device.

12. The system of claim 11 , wherein each storage device of the plurality of storage devices is associated with a failure domain, the failure domain being shared with one or more other storage devices of the plurality of storage devices, wherein storing the regenerated fragment on the replacement storage device comprises: an identification that the replacement storage device is associated with the same failure domain as another storage device of the first subset; as well as It is determined that the regenerated segment and the segment associated with the first data and stored on the other storage device of the first subset are compatible with storage devices in the same failure domain.

13. The system of claim 9, wherein the determination to replace a first storage device comprises a determination that the first storage device is unavailable.

14. The system of claim 9, wherein each storage device of the plurality of storage devices is associated with a failure domain, the failure domain being shared with one or more other storage devices of the plurality of storage devices, and wherein determining the first plurality of sets of the first number of storage devices comprises: Randomly permuting the fault domains; randomizing the storage devices associated with each of the failure domains; merging the arrangement into a list, the list being ordered first by a position of each storage device in the arrangement of storage devices in the same failure domain, and second by a position of the storage device's failure domain in the arrangement of failure domains; as well as The list is divided into two or more groups of the first number of storage devices, wherein each group is one of the plurality of first sets of the first number of storage devices.

15. The system of claim 9, wherein determining a first subset of a second number of storage devices from a first one of the first sets of storage devices comprises: determining a first subset of a second number of storage devices from a first one of said first sets of storage devices; determining a second candidate subset of the second number of storage devices from a second first set of the first set of the first number of storage devices; determining the free storage capacity of the first subset and the free storage capacity of the second candidate subset; as well as The first subset of the second number of storage devices is determined based on the free storage capacity of the first subset and the free storage capacity of the second candidate subset.

16. The system according to claim 9, wherein the server is further configured to: receiving third data to be stored; determining a third subset of said second number of storage devices from a second first set of said first sets of storage devices; storing the third data in a fault-tolerant format using the third subset of the second number of storage devices; determining to replace a first storage device in the third subset of the second number of storage devices; as well as In response to the determination to replace the first storage device in the third subset of the second number of storage devices: determining a third replacement storage device from said second one of said first sets of storage devices, wherein said third subset of said second number of storage devices does not include said third replacement storage device; as well as The third data is stored in a fault-tolerant format using the third replacement storage device and the third subset of the second number of storage devices excluding the first storage devices of the third subset of the second number of storage devices.

17. A method for storing data in a fault-tolerant format across a plurality of storage devices, The method comprises: determining a plurality of first sets of storage devices, each of the first sets comprising a first number of storage devices; receiving first data to be stored; determining a first subset of a second number of storage devices from a first one of the first sets of storage devices, wherein the second number is less than the first number; storing the first data in a fault-tolerant format using the first subset of the second number of storage devices; determining a first storage device to replace said first subset of said second number of storage devices; as well as In response to determining to replace the first storage device: determining a replacement storage device from said first one of said first sets of storage devices, wherein said first subset of said second number of storage devices does not include said replacement storage device; as well as The first data is stored in a fault-tolerant format using the replacement storage device and the first subset of the second number of storage devices, excluding the first storage device.

18. The method of claim 17, wherein storing the first data in a fault-tolerant format using the first subset of the second number of storage devices comprises copying the first data on each storage device in the first subset of the second number of storage devices, and Wherein storing the first data in a fault-tolerant format using the replacement storage device and the first subset of the second number of storage devices excluding the first storage device includes copying the first data on the replacement storage device.

19. The method of claim 17, wherein storing the first data in a fault-tolerant format using the first subset of the second number of storage devices comprises: erasure encoding the first data to generate a second number of fragments associated with the first data, and storing each of the second number of fragments on a corresponding storage device of the first subset of the second number of storage devices, and Wherein storing the first data in a fault-tolerant format using the replacement storage device and the first subset of the second number of storage devices, excluding the first storage device, includes regenerating fragments associated with the first data and stored on the first storage device, and storing the regenerated fragments on the replacement storage device.

20. The method of claim 17, further comprising: receiving second data to be stored; determining a second subset of said second number of storage devices from a first one of said first sets of storage devices; storing the second data in a fault-tolerant format using the second subset of the second number of storage devices; determining to replace a first storage device in said second subset of said second number of storage devices; as well as In response to determining to replace the first storage device in the second subset of the second number of storage devices: determining a second replacement storage device from said first one of said first sets of storage devices, wherein said second subset of said second number of storage devices does not include said second replacement storage device; as well as The second data is stored in a fault-tolerant format using the second replacement storage device and the second subset of the second number of storage devices excluding the first storage devices in the second subset of the second number of storage devices.

Images