Distributed object store, method and computer readable medium for synchronizing data stores

Abstract

The present disclosure provides a method for synchronizing a plurality of data stores of a distributed object store, the method comprising: causing a first data store of the data stores to become online after having been offline; selecting a second data store of the data stores, the second data store being online when the first data store is offline; reading an entry from a follow-up log by the first data store; and performing a data modification operation corresponding to the entry on the first data store to synchronize the plurality of data stores. A distributed object store and a non-transitory computer readable medium are also provided.

Description

TECHNICAL FIELD

[0001] One or more aspects of embodiments of the present disclosure relate generally to a method of synchronizing data stores of a distributed object store, and a distributed object store configured to perform the method. BACKGROUND

[0002] Distributed object stores achieve performance, scalability, and data resiliency through data replication and through balancing data distribution. For example, data is distributed across different data stores (e.g., disks) of underlying clusters of data stores of a distributed system.

[0003] A drawback of balancing data distribution in conventional distributed object stores is that any disruption or failure of a data store in a cluster generally causes a data distribution algorithm in the distributed object store to be re-run. Running the distribution algorithm can move affected data corresponding to the failed data store, thereby restoring data resiliency and balanced distribution in the underlying cluster. Moving the affected data can cause poor performance of the distributed object store, as system performance can be negatively impacted during data redistribution operations due to additional background input / output (I / O) performed to move and replicate data to restore data resiliency. SUMMARY

[0004] Embodiments set forth herein provide improvements to a method of synchronizing data stores of a distributed object store after bringing data stores of a distributed system back online.

[0005] According to one embodiment of the present disclosure, a method of synchronizing a plurality of data stores of a distributed object store is provided, the method comprising: bringing a first data store of the data stores online after having been offline; selecting a second data store of the data stores, the second data store being online when the first data store was offline; reading, by the first data store, an entry from a catch-up log; and performing, on the first data store, a data modification operation corresponding to the entry to synchronize the plurality of data stores.

[0006] The method can further comprise bringing the first data store offline in accordance with a scheduled maintenance operation prior to bringing the first data store online.

[0007] The method can further comprise storing the catch-up log on a plurality of nodes of a cluster underlying the distributed object store.

[0008] The follow-up log can be an unordered collection of entries representing respective data modification operations to be performed on the first data store.

[0009] The entries can represent a key corresponding to a data object and a type of the data modification operation to be performed on the data object.

[0010] The entries can include a delete entry, the data modification operation can include a delete operation to delete an object, and the method can further include checking, by the first data store, the second data store to determine whether the object is contained in the second data store, discarding, by the first data store, the delete entry when the object is contained in the second data store, and performing, by the first data store, the delete operation to delete the object from the first data store when the object is not present in the second data store.

[0011] The entries can include a write entry, the data modification operation can include a write operation to write an object, and the method can further include checking, by the first data store, the second data store to determine whether the object is contained in the second data store, discarding, by the first data store, the write entry when the object is not present in the second data store, and performing, by the first data store, the write operation to write the object to the first data store when the object is in the second data store.

[0012] According to another embodiment of the disclosure, there is provided a distributed object store comprising a plurality of data stores, the distributed object store being configured to synchronize by having a first data store among the data stores become online after having been offline, electing one or more stores among the data stores, the elected stores being online when the first data store is offline (electing a second data store among the data stores, the second data store being online when the first data store is offline), reading, by the first data store, an entry from a follow-up log, and performing, by the first data store, a data modification operation corresponding to the entry to synchronize the plurality of data stores.

[0013] The distributed object store can be further configured to synchronize by having the first data store go offline according to a scheduled maintenance operation before having the first data store become online.

[0014] The distributed object store can be further configured to synchronize by storing the follow-up log on a plurality of nodes of a cluster underlying the distributed object store.

[0015] The follow-up log can be an unordered collection of entries representing respective data modification operations to be performed on the first data store.

[0016] The entries can represent a key corresponding to a data object and a type of the data modification operation to be performed on the data object.

[0017] The entries can include a delete entry, the data modification operation can include a delete operation to delete an object, and the distributed object store can be further configured to synchronize by the first data store checking the second data store to determine whether the object is contained in the second data store, discarding the delete entry by the first data store when the object is contained in the second data store, and performing the delete operation by the first data store to delete the object from the first data store when the object is not present in the second data store.

[0018] The entries can include a write entry, the data modification operation can include a write operation to write an object, and the distributed object store can be further configured to synchronize by the first data store checking the second data store to determine whether the object is contained in the second data store, discarding the write entry by the first data store when the object is not present in the second data store, and performing the write operation by the first data store to write the object to the first data store when the object is in the second data store.

[0019] According to yet another embodiment of the present disclosure, there is provided a non-transitory computer readable medium having computer code embodied thereon for implementing a method of synchronizing a distributed object store comprising a plurality of data stores when executed on a processor, the method comprising bringing a first data store of the data stores online after having been offline, selecting a second data store of the data stores, the second data store being online when the first data store is offline, reading an entry from a follow-up log by the first data store, and performing a data modification operation corresponding to the entry on the first data store to synchronize the plurality of data stores.

[0020] The computer code, when executed by the processor, can further implement the method of synchronizing the distributed object store by bringing the first data store offline according to a scheduled maintenance operation before bringing the first data store online.

[0021] The computer code, when executed by the processor, can also implement a method of synchronizing the distributed object store by storing the follow-up log on multiple nodes of a cluster underlying the distributed object store.

[0022] The follow-up log can be an unordered collection of entries representing respective data modification operations to be performed on the first data store.

[0023] The entries can represent keys corresponding to data objects and types of the data modification operations to be performed on the data objects.

[0024] The entries can include a deletion entry, the data modification operation can include a delete operation to delete an object, and the computer code, when executed by the processor, can also implement a method of synchronizing the distributed object store by checking, by the first data store, the second data store to determine whether the object is contained in the second data store; discarding, by the first data store, the deletion entry when the object is contained in the second data store; and performing, by the first data store, the delete operation to delete the object from the first data store when the object is not present in the second data store.

[0025] Accordingly, the distributed object store of embodiments of the present disclosure can achieve improved performance by avoiding unnecessary data migration and / or replication after a data store of a distributed system is brought back online. BRIEF DESCRIPTION OF DRAWINGS

[0026] The above and / or other aspects will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0027] Figure 1 Illustrates a conceptual diagram of a follow-up log according to one or more embodiments of the present disclosure.

[0028] Figure 2 Illustrates log entries in a follow-up log corresponding to different dependent data modification operations respectively handled by different nodes in an underlying cluster of a distributed object store according to one or more embodiments of the present disclosure.

[0029] Figure 3 Illustrates parallel synchronization of a distributed object store after a data store is brought back online by reading entries of a follow-up log from different nodes in an underlying cluster of the distributed object store according to one or more embodiments of the present disclosure.

[0030] [LEGEND]

[0031] 100, 100a, 100b: follow-up log;

[0032] 101: key-value pair;

[0033] 110, K1: key;

[0034] 120: operation code;

[0035] 200: cluster;

[0036] 210: node / node N;

[0037] 220: node / node 1;

[0038] 230: affected data store;

[0039] 240: reference data store;

[0040] T1: first time / time;

[0041] T2: second time / time. DETAILED DESCRIPTION

[0042] The features of the present inventive concept and the method of implementing the same can be understood more readily by reference to the following detailed description of embodiments and the accompanying drawings. In the following detailed description, reference will be made to the accompanying drawings, which form a part of the detailed description. The illustrative embodiments described in the detailed description, scope of the inventive concept and their equivalents will be described and explained with additional specificity and detail through the use of the accompanying drawings. The described embodiments are merely examples of the inventive concept and should not be considered limiting of the scope of the inventive concept. Indeed, various modifications of the inventive concept, in addition to those described herein, will become apparent to those skilled in the art from the description contained herein. Such modifications are intended to fall within the scope of the inventive concept. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this inventive concept belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the inventive concept. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. In addition, it is to be understood that the phraseology or terminology employed herein, and not otherwise specifically defined, is for the purpose of description only and not of limitation. Any use of section headings is intended to aid reading the technical description and is not to be interpreted as limiting.

[0043] A distributed object store includes a cluster of independent data stores (e.g., disks). Each of the data stores is capable of storing data or objects thereon. Generally, a distributed object store will attempt to keep the various data stores evenly balanced. By keeping the data stores relatively balanced, a distributed object store can prevent one of the data stores from running out of space while other data stores have available space, thereby avoiding access hot spots.

[0044] Some of the data stores of a distributed object store will store replica data. That is, one data store can store objects that are replicated on one or more other data stores. Thus, a distributed object store will allow data replication to allow for the failure of one of the data stores.

[0045] During a planned maintenance operation (e.g., when taking a data store offline to upgrade the data store), the remaining data stores of the distributed object store will remain operational. However, commands for performing data modification operations involving the offline data store can continue to be processed while the data store is offline. For example, the distributed object store can receive new write commands for performing writes to the offline data store during the maintenance operation.

[0046] If the resilience requirements corresponding to the objects intended to be written to the offline data store require the objects to be written to two different data stores, the distributed object store will ensure that the objects are written to two other online data stores. This can result in a significant amount of work for the distributed object store, as a relatively large amount of data can be moved between the various data stores of the distributed object store during the balancing.

[0047] In a conventional distributed object store, the amount of data movement depends on the data distribution algorithm used by the distributed object store to control the distribution of data among the cluster of data stores. In the best case, the amount of data movement will be at least equal to the amount of data affected by the interruption of the offline data store.

[0048] However, the amount of data movement will typically be greater than the amount of data affected by the interruption, as the data distribution algorithm will generally also attempt to achieve a balanced distribution of data among the cluster of data stores. That is, in addition to the data affected by the data store interruption, additional data can also be moved across the cluster during activities directed by the data distribution algorithm.

[0049] Thus, a conventional distributed object store redistributes data among the cluster to restore resilience and balanced distribution. The amount of time to complete the data redistribution will be proportional to the amount of data being redistributed. As a distributed object store scales, planned interruptions of individual components (e.g., maintenance of a data store) will occur while the distributed object store remains operational. However, triggering a data redistribution operation to synchronize individual data stores across the cluster of the distributed object store after a planned interruption generally will suppress the performance of the distributed object store immediately after any planned maintenance operation.

[0050] Embodiments of the present disclosure provide a high performance distributed object store by avoiding data redistribution as the primary means of synchronizing data across the cluster, while also being able to restore resilience and balanced data distribution after a planned interruption.

[0051] Embodiments of the present disclosure also provide a method that efficiently tracks changes to affected data stores during a planned outage, while using reduced or minimal amount of information, and efficiently synchronizes them when the affected stores come back online after the planned outage.

[0052] Accordingly, embodiments of the present disclosure improve memory technology by providing a high performance distributed object store that enables high degree of concurrency and parallelism in data replication operations by providing a solution that is available during maintenance operations in a distributed system.

[0053] Figure 1 A conceptual diagram illustrating a follow-up log is depicted in accordance with one or more embodiments of the present disclosure.

[0054] Referring to Figure 1 Embodiments provide a mechanism referred to herein as a "follow-up log." The follow-up log 100 is a log of entries that respectively represent data modification operations to be performed on an underlying data store (e.g., an affected data store or data store "A," which will be described below with reference to Figure 2 and Figure 3 Embodiments provide a mechanism that enables the underlying data store to synchronize with other data stores in the cluster to restore data resiliency and consistency when the underlying data store is offline (e.g., during a planned maintenance operation).

[0055] To enable the use of the follow-up log 100, embodiments provide a mechanism that records data modification operations that occur during a planned outage with zero contention parallelism. Embodiments also provide a mechanism that processes entries in the follow-up log 100 in parallel and scales linearly with the size of the cluster, thereby enabling efficient bringing of data stores back online once the planned outage is respectively completed.

[0056] As Figure 1As shown, the distributed object store of the present embodiments uses key-value pairs 101 to organize data, where the key 110 of the key-value pair 101 is an object identifier, and the operation code 120 of the operation type that includes the key-value pair 101 is opaque data associated with the object identifier. As will be further set forth below, the follow-up log 100 does not record all data, but rather the operation type on the key. That is, one aspect of the disclosed embodiments is that the follow-up log 100 does not record data, but rather simply records the key 110 and the operation code 120. The replay of the follow-up log 100 depends on reading data from the commodity, online store (i.e., the relevant store that was online during the planned outage), and then updating the store that came back online (i.e., after being offline during the planned outage). This allows the use of a small amount of memory that can keep track of a large number of changes, which generally increases the amount of downtime that can be supported.

[0057] During a data store outage, a follow-up log 100 is generated for tracking data modification operations that occurred during the outage and that involve the offline data store. For example, if a data entry on the offline data store is to be deleted or modified, when the data store comes back online, the data store can simply read or replay the entries of the follow-up log 100 to synchronize with the remaining data stores of the cluster.

[0058] The follow-up log 100 uses only two pieces of information - the key 110 of the data being modified and the operation type of the data modification operation (e.g., write, delete, etc.). Thus, the size of the entries in the follow-up log 100 can be relatively small, allowing a large number of entries to be recorded without an undue increase in memory or storage requirements. Thus, the small follow-up log entries also allow for an increase in the time of the planned outage of the data store.

[0059] In the present embodiments, the follow-up log 100 is organized to allow multiple hardware threads to record entries (e.g., data modification operations) into the follow-up log 100 in parallel.

[0060] A cluster of distributed object stores can include multiple nodes (e.g., the nodes 210, 220 set forth below with respect to Figure 2 and Figure 3 Each node in the cluster can have an independent follow-up log 100, and each independent follow-up log 100 can be organized to have independent partitions for each respective hardware thread. Thus, multiple writes in the cluster can record entries in the follow-up log 100 in parallel without contention, and the multiple writes thus have zero impact on the performance IO path during a planned outage.

[0061] When the data stores come back online (e.g., after completing the scheduled maintenance operation), the distributed object store can be synchronized. Once the data stores are online (e.g., after completing the scheduled outage), the distributed object store is synchronized by processing the follow-up log 100 to synchronize the affected data store with the rest of the cluster.

[0062] To synchronize the affected data store, a reference data store is selected. The reference data store is a data store that remains online during the scheduled outage of the affected data store and stores data used to synchronize the affected data store.

[0063] Once the reference data store is selected, the entries from the follow-up log 100 can be read or replayed. When reading the entries of the follow-up log 100, the affected data store can access the data corresponding to the key 110 included in the entry from the reference data store. Then, the distributed object store can perform the data modification operation represented by the entry in the follow-up log 100 on the affected data store.

[0064] To enable parallelism during both the logging phase for generating the follow-up log 100 and the replay phase for performing the data modification operations corresponding to the entries of the follow-up log 100, the present embodiment provides a mechanism for handling ordering constraints.

[0065] Figure 2 A log entry in the follow-up log 100 corresponding to a different related data modification operation processed by a different node in the underlying cluster of the distributed object store is illustrated in accordance with one or more embodiments of the present disclosure.

[0066] Referring to Figure 2 In the present example, the follow-up log 100 is created to record data modification operations performed on the affected data store (data store "A") 230 that is currently offline due to scheduled maintenance. In the present example, a data modification operation on the affected data store 230 is received in "node N" 210 and another data modification operation on the affected data store 230 is received in "node 1" 220.

[0067] Traditionally, if the affected data store 230 replays the entries in the journal 100 out of order, the recovery operation can inadvertently delete valid objects in the affected data store 230. For example, during a planned maintenance operation, there can first be an entry corresponding to a delete operation for a distributed object stored on the affected data store 230 added to the journal 100b, and thereafter there can be another entry corresponding to a write operation to write the object back into the affected data store 230 added to the journal 100a.

[0068] That is, at a first time Tl, an operation to delete the key Kl arrives at a given node (node N) 210 and is recorded in the journal 100b stored on the node 210. Then, at a later second time T2, an operation to add the key Kl arrives at a different node (node 1) 220 and is recorded in the journal 100a stored on the other node 220.

[0069] Thus, if upon reactivation of the affected data store 230, the entry corresponding to the write operation at the later second time T2 is first read from the journal, the key Kl will be populated in the affected data store 230. However, when the entry corresponding to the delete operation that occurred at the earlier first time Tl is later read from the journal 100a, this key Kl will be deleted, resulting in a valid entry in the affected data store being deleted. This can result in data loss due to the order sequencing constraint between the operations performed at times Tl and T2.

[0070] Figure 3 According to one or more embodiments of the present disclosure, parallel synchronization of a distributed object store is performed by reading journal entries from different nodes in the underlying cluster of the distributed object store after bringing the data store online.

[0071] Referring to Figure 3 Embodiments of the present disclosure avoid unnecessary data movement (e.g., from the reference data store 240 to the affected data store 230) by avoiding order sequencing constraints that would otherwise require replaying journal entries in a sequential, time-ordered manner, and thus would inhibit parallel recovery of the affected data store 230.

[0072] That is, embodiments of the present disclosure avoid in-order replay, which would otherwise require the ability to sort entries in the journal 100 based on a reference time in the cluster 200, and the ability to sequentially replay the sorted list of entries. Such in-order replay would prevent synchronization of the affected data store 230 from being completed in parallel in the cluster 200.

[0073] Instead, the present embodiments solve the problem caused by the ordering constraint by effectively making the follow log 100 an unordered collection of entries representing data modification operations. By having an unordered follow log 100, the present embodiments are able to record operations in parallel during the planned maintenance and recover in parallel during the synchronization of the data stores, thereby unlocking the scalability and performance for these operations in a distributed object store.

[0074] In the present embodiments, the ordering constraint between different entries in the follow log 100 is eliminated as follows.

[0075] After determining the data modification operations that will be performed according to the entries in the follow log 100, the distributed object store can synchronize the data modification operations for a particular key 110 by using locks specific to the key 110 in the affected data store 230. Thus, only one thread can modify the data corresponding to the key 110.

[0076] The affected data store 230 can then check to determine if the object has been erased from the reference data store 240. That is, the affected data store 230 can make a pass of the reference data store 240 to determine if the object corresponding to the data modification operation was deleted from the reference store while the affected data store 230 was offline, and if the deleted object was kept.

[0077] If the entry corresponds to a delete operation, and if the affected data store 230 determines that the object exists in the reference data store 240, no action is taken and the follow log entry corresponding to the object is discarded. That is, by determining that the object exists in the reference data store 240, the affected data store 230 can assume that the object was recreated after the delete operation, and that another entry in the follow log 100 that the affected data store 230 has not read during the replay of the follow log 100 corresponds to a write operation that recreated the object.

[0078] If the affected data store 230 determines that the object does not exist in the reference data store 240, the affected data store 230 can perform the delete / erase operation in the affected data store 230.

[0079] Similarly, if the entry corresponds to a write operation, and if the affected data repository 230 determines that the object does not exist in the reference data repository 240, the follow-up log entry corresponding to the object is not processed but discarded. That is, by determining that the object does not exist in the reference data repository 240, the affected data repository 230 may assume that the object was deleted after the write operation, and that another entry in the follow-up log 100 that the affected data repository 230 has not yet read during the replay of the follow-up log 100 corresponds to the data modification operation that deleted the object.

[0080] If the affected data store 230 determines that the object exists in the reference data store 240 , the affected data store 230 may proceed with the write operation in the affected data store 230 .

[0081] Thus, embodiments of the present disclosure provide improved or optimal data logging for each key-value operation, provide scalable organization of the follow-up log to allow concurrent parallel writes to the follow-up log, enable removal of ordering constraints between related operations to allow replay in any order, and enable concurrent replay from the follow-up log to allow parallel recovery across multiple nodes in a cluster of distributed object repositories.

[0082] Embodiments of the present disclosure enable recording entries in the follow-up log with zero contention, to achieve a relatively small entry size so that millions of operations can be recorded in the follow-up log, thereby allowing for an increased amount of time available for planned outages, using only a small amount of memory or storage dedicated to the follow-up log, and achieving linear scalability while recovering from the follow-up log due to the parallelism in processing the follow-up log.

[0083] In the description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the various embodiments. However, it is apparent that the various embodiments can be practiced without these specific details or with one or more equivalent configurations. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the various embodiments.

[0084] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" "comprising," "has" "having," "includes" "including" and / or "contains" "containing," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0085] When a certain embodiment can be implemented in different ways, a specific process sequence can be executed in a different order from the described order. For example, two consecutively described processes can be executed substantially simultaneously or in an opposite order to the described order.

[0086] Electronic devices or electrical devices and / or any other related devices or components according to embodiments of the present disclosure described herein can be implemented utilizing any suitable hardware, firmware (e.g., an application-specific integrated circuit), software, or combinations thereof. For example, various components of the devices can be formed on one integrated circuit (IC) chip or on separate IC chips. In addition, various components of the devices can be implemented, for example, on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB), or formed on a substrate. Further, various components of the devices can be processes or threads running on one or more computers, executing computer program instructions as a sequence of instructions, and interacting with other system components for performing the various functions described herein. The computer program instructions can be stored in a memory, which can be implemented in a computing device using, for example, standard memory devices, such as random access memory (RAM). The computer program instructions can also be stored in other non-transitory computer readable media such as, for example, a compact disc read only memory (CD-ROM), a flash drive, or the like. Also, those skilled in the art will appreciate that the functionality of various computing devices can be combined or integrated into a single computing device, or the functionality of a particular computing device can be distributed across one or more other computing devices without departing from the spirit and scope of embodiments of the present disclosure.

[0087] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this inventive concept belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and / or the present disclosure and will not be interpreted in an overly literal or overly formal sense unless expressly so defined herein.

[0088] Embodiments have been disclosed herein and, although in a specific form, are used in a generic and illustrative sense, or to be construed in a generic and illustrative sense, and are not for limiting purposes. In some instances, features, attributes and / or elements that are described in connection with a particular embodiment can be used alone or in combination with features, attributes and / or elements described in connection with another embodiment, unless otherwise expressly stated, e.g., to the contrary. Accordingly, one of ordinary skill in the art will recognize that the foregoing and various other changes in form and detail can be made without departing from the spirit and scope of the disclosure as set forth in the above claims, wherein the functional equivalents of the claims are included herein.

Claims

1. A method of synchronizing a plurality of data stores of a distributed object store, the method comprising: bringing a first data store of the plurality of data stores online after having been offline; selecting a second data store of the plurality of data stores, the second data store being online when the first data store is offline; reading, by the first data store, an entry from a follow-up log; and performing, on the first data store, a data modification operation corresponding to the entry to synchronize the plurality of data stores, wherein performing, on the first data store, the data modification operation corresponding to the entry comprises: if the data modification operation comprises a delete operation to delete an object, performing the delete operation on the first data store to delete the object from the first data store when the object is not contained in the second data store; and if the data modification operation comprises a write operation to write the object, performing the write operation on the first data store to write the object to the first data store when the object is contained in the second data store. bringing the first data store online prior to bringing the first data store offline according to a scheduled maintenance operation.

2. The method of claim 1, further comprising: storing the follow-up log on a plurality of nodes of an underlying cluster of the distributed object store.

3. The method of claim 1, further comprising:

4. The method of claim 1, wherein the follow-up log is an unordered set of entries representing respective data modification operations to be performed on the first data store.

5. The method of claim 1, wherein the entry indicates a key corresponding to a data object and a type of the data modification operation to be performed on the data object.

6. The method of claim 1, wherein the entry comprises a delete entry, the method further comprising: checking, by the first data store, the second data store to determine whether the object is contained in the second data store; and discarding, by the first data store, the delete entry when the object is contained in the second data store.

7. The method of claim 1, wherein the entry comprises a write entry, the method further comprising: checking, by the first data store, the second data store to determine whether the object is contained in the second data store; and discarding, by the first data store, the write entry when the object is not contained in the second data store.

8. A distributed object store comprising a plurality of data stores, the distributed object store configured to be synchronized by: bringing a first data store of the plurality of data stores online after having been offline; selecting a second data store of the plurality of data stores, the second data store being online when the first data store is offline; reading, by the first data store, an entry from a follow-up log; and performing, on the first data store, a data modification operation corresponding to the entry to synchronize the plurality of data stores, wherein performing, on the first data store, the data modification operation corresponding to the entry comprises: ​ ​ ​ ​ if the data modification operation comprises a delete operation to delete an object, performing the delete operation on the first data store to delete the object from the first data store when the object is not contained in the second data store; and if the data modification operation comprises a write operation to write the object, performing the write operation on the first data store to write the object to the first data store when the object is contained in the second data store.

9. The distributed object store of claim 8, wherein the distributed object store is further configured to synchronize by: taking the first data store offline in accordance with a scheduled maintenance operation before bringing the first data store online.

10. The distributed object store of claim 8, wherein the distributed object store is further configured to synchronize by: storing the follow-up log on a plurality of nodes of an underlying cluster of the distributed object store.

11. The distributed object store of claim 8, wherein the follow-up log is an unordered collection of entries representative of respective data modification operations to be performed on the first data store.

12. The distributed object store of claim 8, wherein the entries represent a key corresponding to a data object and a type of the data modification operation to be performed on the data object.

13. The distributed object store of claim 8, wherein the entries comprise delete entries, and wherein the distributed object store is further configured to synchronize by: checking, by the first data store, the second data store to determine whether the object is contained in the second data store; and discarding, by the first data store, the delete entry when the object is contained in the second data store.

14. The distributed object store of claim 8, wherein the entries comprise write entries, and wherein the distributed object store is further configured to synchronize by: checking, by the first data store, the second data store to determine whether the object is contained in the second data store; and discarding, by the first data store, the write entry when the object is not contained in the second data store.

15. A non-transitory computer readable medium having computer code embodied thereon for implementing a method of synchronizing a plurality of data stores of a distributed object store system, the method comprising: bringing a first data store of the plurality of data stores online after having been offline; selecting a second data store of the plurality of data stores, the second data store being online when the first data store is offline; reading, by the first data store, an entry from a follow-up log; and performing, on the first data store, a data modification operation corresponding to the entry to synchronize the plurality of data stores. ​ wherein performing the data modification operation corresponding to the entry on the first data store comprises: if the data modification operation comprises a delete operation to delete an object, performing the delete operation on the first data store to delete the object from the first data store when the object is not contained in the second data store; and if the data modification operation comprises a write operation to write the object, performing the write operation on the first data store to write the object to the first data store when the object is contained in the second data store.

16. The non-transitory computer-readable medium of claim 15, wherein the computer code, which when executed by the processor, further implements the method of synchronizing the plurality of data stores of the distributed object store by: taking the first data store offline in accordance with a scheduled maintenance operation prior to bringing the first data store online.

17. The non-transitory computer-readable medium of claim 15, wherein the computer code, which when executed by the processor, further implements the method of synchronizing the plurality of data stores of the distributed object store by: storing the follow-up log on a plurality of nodes of an underlying cluster of the distributed object store.

18. The non-transitory computer-readable medium of claim 15, wherein the follow-up log is an unordered set of entries representative of respective data modification operations to be performed on the first data store.

19. The non-transitory computer-readable medium of claim 15, wherein the entry indicates a key corresponding to a data object and a type of the data modification operation to be performed on the data object.

20. The non-transitory computer-readable medium of claim 15, wherein the entry comprises a delete entry, and wherein the computer code, which when executed by the processor, further implements the method of synchronizing the plurality of data stores of the distributed object store by: checking, by the first data store, the second data store to determine whether the object is contained in the second data store; discarding, by the first data store, the delete entry when the object is contained in the second data store.

Images