A method, system, and storage medium for recovery of metadata for a distributed object store

Abstract

The application discloses a kind of distributed object storage metadata recovery method, system and storage medium.The method comprises the following steps: generating incremental log when carrying out metadata operation;For each object needing recovery, the master OSD traverses the incremental log, finds all incremental information of each key of each object needing recovery;The master OSD determines the last incremental information of each key of each object needing recovery in all incremental information of each key of each object needing recovery;When the key is missing from the OSD, the last incremental information corresponding operation is executed for each key of each object needing recovery;When the key is missing from the master OSD, the last incremental information corresponding operation is executed for each key of each object needing recovery.By the application, incremental recovery can be used when data recovery is carried out, avoiding full traversal of all key-value pairs, reducing data recovery time, greatly improving the recovery speed and efficiency of metadata.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and in particular to a method, system, and storage medium for recovering metadata from distributed object storage. Background Technology

[0002] In distributed storage systems, metadata management and storage are crucial for ensuring efficient read and write performance. In recent years, with the rapid development of big data, the LSM-Tree (Log-Structured Merge-Tree) structure has been widely adopted due to its high write performance. Currently, many systems use LSM tree technology to store metadata. The core idea of ​​LSM-Tree is to improve write performance through sequential writes; LSM-Tree is not a tree-like data structure, but merely a storage structure. A typical example is RoCSDB, which also uses LSM-Tree technology to store system metadata and pushes the metadata progressively to different levels. Data is first placed in level L0. When each level reaches a certain threshold, the data is moved to a deeper level, i.e., L0->L1, L1->L2, and so on. Simultaneously, each level has its own fixed storage files (usually multiple files per level) to store these key-value pairs.

[0003] For distributed storage systems, storing data metadata in ROCSCDB is a common choice. To ensure data reliability, metadata is typically stored across multiple replicas, usually three copies. Since a cluster can contain billions or even tens of billions of data entries, a placement group (PG) concept is designed. All written data is hashed and distributed across various placement groups. The number of placement groups is typically kept to a few thousand, which avoids hotspot effects and facilitates effective data management.

[0004] Meanwhile, each placement group typically consists of object storage devices (OSDs) on different nodes. Each OSD usually corresponds to one disk, and each OSD runs an independent RocksDB instance. For example... Figure 1 As shown, after receiving a request, the master OSD will write the metadata to the corresponding ROCCSDB and push the data to other slave OSDs in the same PG. This can effectively ensure the reliability of the data.

[0005] However, when one of the PG replicas is lost or its content is inconsistent with that of the other replicas, the system's data recovery process will be triggered. The data recovery process involves comparing the logs recorded by each OSD to calculate the missing data set of the other placement group. Based on this set, each piece of data is copied from the complete metadata node to the rocksdb on the missing node.

[0006] The current metadata information is recorded in RocksDB. For ease of management, when operating on a directory or bucket, a unique and fixed prefix related to the directory or bucket name is used as the storage key, and the value is the metadata information to be stored.

[0007] Taking object storage as an example, the metadata of each object in a bucket is distributed across multiple bucket index shards according to the hash. Each index shard file stores tens of thousands of key-value pairs, and these key-value pairs have the same prefix.

[0008] To ensure the atomicity of I / O, metadata falling on the same index shard is queued and processed uniformly within the PG. Scheduling is based on the index shard as the smallest unit, so... Figure 2 As shown, when an index shard is undergoing data recovery, metadata requests falling on the same index shard must wait for the index shard recovery to complete before the metadata write operation can be performed. If the index shard recovery speed is too slow, it will block I / O indefinitely, and may even cause a large number of client requests to fail.

[0009] The recovery speed of each index shard is positively correlated with the key-value pairs (meta-information) associated with each shard. This is because the index shard data is stored in RocksDB, and this data is not concentrated in a single storage file (SST file) within RocksDB, but rather scattered across various levels and within each level's SST file. Therefore, we cannot recover index shards by directly copying SST files. Instead, we must traverse all key-value pairs of the complete index shard and then fully rewrite them to the missing index shard.

[0010] Figure 3 This demonstrates the process of restoring data from the main OSD. Figure 3 As can be seen, when restoring the index file, it is necessary to traverse all the keys of the existing index objects, delete them all, and then rewrite the keys read from other OSDs.

[0011] Suppose an index file has 200,000 keys. If a node loses power and a degraded write occurs in a certain product group (PG), for example, only two copies of the data are written in a 3-replica group. When an OSD is re-added to this PG, even if only one metadata item (one key) is missing, the above process must be repeated: the local OSD iterates through all 200,000 keys, deletes them all, reads the remaining 200,000 keys from the peer, and rewrites them locally. This results in very low recovery efficiency. More seriously, this process can easily cause excessively deep RocksDB I / O iterations, leading to OSD process timeouts and crashes. Summary of the Invention

[0012] This application provides a method, system, and storage medium for recovering metadata from distributed object storage, aiming to solve the technical problem that existing data recovery processes require traversing all keys of existing index objects, resulting in low recovery efficiency.

[0013] Firstly, a method for recovering metadata from distributed object storage, applied to a cloud service architecture, the method comprising:

[0014] S1 generates incremental logs when performing metadata operations;

[0015] S2, the main OSD calculates the data missing status of the placement group based on the full log;

[0016] S3, When performing data recovery of the target index file, the main OSD traverses the incremental log for each object that needs to be recovered, and finds all incremental information for each key of each object that needs to be recovered;

[0017] S4, the master OSD determines the last incremental information of each key of each key of each object that needs to be recovered from all incremental information;

[0018] S5, when the master OSD calculates that the slave OSD is missing a key, the master OSD sends the last incremental information of each key of each object that needs to be recovered to the slave OSD; the slave OSD performs the operation corresponding to the last incremental information of each key of each object that needs to be recovered based on the last incremental information of each key.

[0019] S6. When the master OSD calculates that the master OSD is missing a key, the master OSD performs the operation corresponding to the last incremental information of each key of each object that needs to be recovered, based on the last incremental information of each key of each object that needs to be recovered.

[0020] Optionally, in the above scheme, the generation of incremental logs includes:

[0021] Receive metadata operation requests initiated by the client;

[0022] After the placement group processes the metadata operation request, it records the incremental information corresponding to the metadata operation request in the log to form an incremental log.

[0023] Write the incremental log to disk.

[0024] Optionally, in the above scheme, the incremental information includes the operation type, the name of the key, and the version number.

[0025] Optionally, in the above scheme, step S5 includes:

[0026] The master OSD packages the operations corresponding to the last incremental information of each key of each object that needs to be recovered into a data structure based on the last incremental information of each key of each object that needs to be recovered, and sends the data structure to the slave OSD.

[0027] The data structure is parsed from the OSD, and corresponding operations are performed on each key of each object that needs to be recovered based on the parsing results.

[0028] In the above scheme, optionally, the data structure includes a mapping structure omap_entries for performing set operations and a set keys_to_remove for performing remove operations.

[0029] Optionally, in the above scheme, the method further includes:

[0030] When performing data recovery on the target index file, the metadata is written to a temporary index file;

[0031] After data recovery is complete, suspend IO and merge the target index file and temporary index file.

[0032] Secondly, a distributed object storage metadata recovery system, set up in a cloud service architecture, includes:

[0033] The incremental log generation module is used to generate incremental logs when performing metadata operations;

[0034] The data missing situation calculation module is used by the main OSD to calculate the data missing situation of the placement group based on the full log;

[0035] The incremental log traversal module is used to, when performing data recovery of the target index file, allow the main OSD to traverse the incremental log for each object that needs to be recovered, and find all incremental information for each key of each object that needs to be recovered.

[0036] The incremental information determination module is used by the main OSD to determine the last incremental information of each key of each object that needs to be recovered from all the incremental information of each key of each object that needs to be recovered;

[0037] The OSD recovery module is used when the master OSD calculates that the slave OSD is missing a key. The master OSD sends the last incremental information of each key of each object that needs to be recovered to the slave OSD. The slave OSD then performs the corresponding operation for each key of each object that needs to be recovered based on the last incremental information of each key.

[0038] The master OSD recovery module is used when the master OSD calculates that a key is missing. The master OSD then performs the operation corresponding to the last incremental information of each key of each object that needs to be recovered, based on the last incremental information of each key.

[0039] Thirdly, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0040] Fourthly, a computer program product comprising a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the method described in any one of the first aspects.

[0041] Compared with the prior art, this application has at least the following beneficial effects:

[0042] The method provided in this application introduces incremental logs, which record incremental logs during metadata writing and deletion to track changes to each key (such as writing and deletion). During data recovery, the logs are replayed to determine the latest state of each key. This allows for incremental recovery of the index file, focusing only on missing key-value pairs, avoiding a full traversal of all key-value pairs, minimizing repetitive I / O operations, reducing data recovery time, and significantly improving the speed and efficiency of metadata recovery. Furthermore, in data recovery scenarios where I / O timeouts are less likely, this method reduces significant I / O overhead during data recovery. It also solves the problem of I / O blocking during index shard recovery, which can even cause client service interruptions / failures. Attached Figure Description

[0043] Figure 1 This is a diagram illustrating OSD data operations.

[0044] Figure 2 This is a schematic diagram of the index shard data recovery process;

[0045] Figure 3 A schematic diagram of the primary OSD data recovery process;

[0046] Figure 4 A flowchart illustrating a method for recovering metadata from a distributed object storage system, provided as an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of an incremental log recording method in one embodiment of this application;

[0048] Figure 6 This is a schematic diagram of incremental logs in one embodiment of this application;

[0049] Figure 7 This is a block diagram of the module architecture of a distributed object storage metadata recovery system provided in one embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] In the description of this application, unless otherwise stated, expressions such as "comprising," "including," and "having" also mean "not limited to" (certain units, components, materials, steps, etc.).

[0052] In one embodiment, such as Figure 4 As shown, a method for recovering metadata from distributed object storage is provided. This method is applied to a cloud service architecture and includes:

[0053] S1 generates incremental logs when performing metadata operations;

[0054] S2, the main OSD calculates the data missing status of the placement group based on the full log;

[0055] S3, when performing data recovery of the target index file, the main OSD traverses the incremental log for each object that needs to be recovered, and finds all the incremental information of each key of each object that needs to be recovered;

[0056] S4, the master OSD determines the last incremental information of each key of each key of each object that needs to be recovered from all incremental information;

[0057] S5, when the master OSD calculates that the slave OSD is missing a key, the master OSD sends the last incremental information of each key of each object that needs to be recovered to the slave OSD; the slave OSD performs the operation corresponding to the last incremental information of each key of each object that needs to be recovered based on the last incremental information of each key.

[0058] S6. When the master OSD calculates that the master OSD is missing a key, the master OSD performs the operation corresponding to the last incremental information of each key of each object that needs to be recovered, based on the last incremental information of each key of each object that needs to be recovered.

[0059] Furthermore, such as Figure 5 As shown, the recording methods of the incremental log pglog include:

[0060] Receive metadata operation requests initiated by the client;

[0061] After the placement group processes the metadata operation request, it records the incremental information corresponding to the metadata operation request in the log to form an incremental log.

[0062] Write incremental logs to disk.

[0063] The incremental information includes the operation type, key name, and version number. The operation type could be deletion or a write operation; the final result will be recorded in the pglog. For details on pglog formation, please refer to [link to pglog documentation]. Figure 6 .

[0064] Further, step S5 includes:

[0065] The master OSD packages the operations corresponding to the last incremental information of each key of each object that needs to be recovered into a data structure based on the last incremental information of each key of each object that needs to be recovered, and sends the data structure to the slave OSD.

[0066] The data structure is parsed from the OSD, and corresponding operations are performed on each key of each object that needs to be recovered based on the parsing results.

[0067] The data structure includes a mapping structure omap_entries for performing set operations and a set keys_to_remove for performing remove operations.

[0068] Furthermore, the method also includes:

[0069] When performing data recovery on the target index file, the metadata is written to a temporary index file;

[0070] After data recovery is complete, suspend IO and merge the target index file and temporary index file.

[0071] In the incremental data recovery scheme described above, the key lies in effectively identifying the data items (key-value pairs) that need to be recovered and performing the corresponding operations. The following is a detailed explanation of this process:

[0072] After writing to the incremental log, how do we determine how many keys need to be recovered? When repairing an object (OBJ), the system retrieves the keywords and their corresponding operation records from the incremental log (pglog). The record format is as follows:

[0073] {key,{e'v,operation}}

[0074] Each record contains a key, its version number, and the operation type (such as set or rm).

[0075] For each object requiring repair, the sender iterates through its associated pglog records, searching backwards. This is because it aims to find the last operation on that key to determine the current state. For example:

[0076] Sender:

[0077] The following records were obtained from pglog when fixing obj1:

[0078] {key0,{10'11,set}{11'14,rm}}

[0079] {key4,{11'15,rm}}

[0080] Then iterate through each key, searching from the end to the beginning. When you find the first record, if it is an rm, then the key needs to perform an rm operation; if it is a set, then the key needs to perform a set operation.

[0081] For key0, the last two records represent a set operation (first) and an rm operation (second). Because the last record is an rm operation, an rm operation needs to be performed during recovery. For key4, there is only one record, representing an rm operation. Therefore, an rm operation also needs to be performed during recovery. In this way, the sending end can determine which keys need to have rm operations performed during the recovery process.

[0082] For obj1, the PushOp package contains the following data:

[0083] map<string,logentry> omap_entries empty

[0084] set <string>keys_to_remove = {key0, key4}

[0085] After analysis, the sending end packages the operations to be executed into a PushOp data structure, including:

[0086] omap_entries: This is used to store the key-value pairs that need to be set. In this example, it is empty because all keys that need to be deleted are rm operations.

[0087] `keys_to_remove`: Stores the keys that need to be deleted. In this example, `keys_to_remove` = `{key0, key4}`.

[0088] The same process is used to repair obj2. After repairing obj2, the records obtained from pglog are as follows:

[0089] {key1,{10'12,rm}{11'16,set}}

[0090] For key1, traversing from back to front, we find that the last record is a set operation, so a set operation needs to be performed during the recovery process.

[0091] The packaged PushOp contains the following data:

[0092] map<string,logentry> omap_entries = {key1, bl} (assuming bl is the value corresponding to key1)

[0093] set <string>keys_to_remove is empty because there are no keys that need to be deleted.

[0094] After receiving a PushOp, the receiving end will perform the following operations:

[0095] For key-value pairs in `omap_entries`, perform a `set` operation. For `obj1`, no `set` operation is needed at this point. For keys in `keys_to_remove`, perform a `remove` operation. For `obj1`, key0 and key4 need to be deleted.

[0096] In other words, the receiving end parses PushOp, performs set operations on the keys and values ​​in omap_entries, and performs rm operations on the keys in keys_to_remove. This avoids restoring the entire index object; only the missing small subset of keys needs to be recovered.

[0097] In simple terms, the metadata recovery process is controlled by the master OSD. A placement group typically has three OSDs: one master OSD and two slave OSDs. The full log is stored on the master OSD, which can use the full log information to calculate the missing keys for all OSDs in the placement group (PG).

[0098] If the master OSD is found to be missing a key based on log calculations, it will actively pull the key from the slave OSD that possesses it, and then write the incremental key into the master OSD. Alternatively, if the slave OSD is missing a key, the master OSD will push the key to the slave OSD, and the slave OSD will write the incremental key upon receiving the information. Therefore, in the scheme proposed in this application, the sending end refers to the master OSD, and the receiving end refers to the slave OSD.

[0099] This incremental recovery mechanism operates only on the missing parts, thus avoiding the need to recover the entire index object. This approach significantly improves recovery efficiency, reduces I / O overhead, and ensures that the system can quickly recover to a consistent state in the event of a failure.

[0100] In the above method, when a key is missing from the OSD, the sending end is responsible for generating and sending a data update request. The sending end extracts the key-value pairs that need to be recovered from the log (pglog) and determines which operations need to be performed based on these records. The sending end reads the log records to determine the latest status of each key (whether a set or rm operation is required). Based on the log, a PushOp is generated, which contains information about the operations to be performed (omap_entries and keys_to_remove). This PushOp is then sent to the receiving end.

[0101] PushOp encapsulates a series of operations, typically including data modification instructions to be executed, such as setting the value of a key (set) and deleting a key (remove). PushOp consists of omap_entries and keys_to_remove. omap_entries contains a set of key-value pairs to be set, and keys_to_remove contains a list of keys to be deleted.

[0102] `omap_entries` is a mapping structure used to store key-value pairs that require `set` operations. Each key corresponds to a log entry, representing the state or value that needs to be set during recovery. When the receiving end parses a `PushOp`, it iterates through this mapping and performs the corresponding `set` operation to update the state of the target object. `keys_to_remove` is a collection used to store keys that require `rm` operations. These keys are extracted from the log and represent items that should be deleted during recovery. When the receiving end parses a `PushOp`, it iterates through this collection and performs the corresponding `rm` operation to ensure that no longer needed keys are removed from the target object.

[0103] The receiving end is responsible for receiving data update requests from the sending end and performing corresponding operations to update its local data storage. The receiving end parses the received PushOp, performs a set operation based on the key-value pairs in omap_entries, and performs an rm operation based on keys_to_remove.

[0104] In summary, the overall idea of ​​the method provided in this application embodiment is to use incremental recovery when recovering data from the index file, only needing to recover the missing key-value values.

[0105] To achieve incremental recovery, incremental logs need to be recorded when metadata is written and deleted. During data recovery, the logs are analyzed and merged, and then replayed. Based on the list of missing metadata in the logs, the logs are pushed to the OSDs that need to be recovered.

[0106] With the advent of the era of big data explosion, the demand for distributed storage systems with massive file storage capabilities is becoming increasingly strong. The modified system using the method provided in this application can achieve the following advantages:

[0107] 1. When a single OSD restarts or the entire system restarts, the overall IOPS performance of the system will not decrease by more than 20% (before the modification, the decrease could reach 80% in some scenarios);

[0108] 2. In data recovery scenarios, where I / O is unlikely to time out;

[0109] 3. Significantly reduced I / O overhead for data recovery;

[0110] 4. Object storage uses a flat storage method for metadata. By optimizing the recovery speed of metadata, it solves the problem of limiting index size and improves the carrying capacity of a single bucket.

[0111] The method provided in this application employs incremental recovery of recorded logs, which greatly improves the recovery performance of metadata. The efficient log parsing and merging method ensures data consistency while also guaranteeing fast data recovery, avoiding repeated writing and deletion of the same data.

[0112] The method provided in this application addresses the problem of using full coverage of all key-value pairs during index sharding recovery, avoiding a large number of repetitive IO operations and reducing data recovery time; it also solves the problem of IO blocking during index sharding recovery, and even causing client service interruption / failure.

[0113] In addition, in the event of data recovery during a failure scenario, metadata can be written to a temporary index first, and then the I / O can be suspended and the two indexes can be merged once the data recovery is complete.

[0114] It should be understood that, although Figure 4 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 4 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0115] In one embodiment, such as Figure 7 As shown, a distributed object storage metadata recovery system is provided, set up in a cloud service architecture. The cloud service architecture has a sender and a receiver. The system includes the following program modules:

[0116] The incremental log generation module 701 is used to generate incremental logs when performing metadata operations;

[0117] The data missing situation calculation module 702 is used by the main OSD to calculate the data missing situation of the placement group based on the full log.

[0118] The incremental log traversal module 703 is used to, when performing data recovery of the target index file, the main OSD traverses the incremental log for each object that needs to be recovered, and finds all incremental information for each key of each object that needs to be recovered.

[0119] The incremental information determination module 704 is used by the main OSD to determine the last incremental information of each key of each object that needs to be recovered from all the incremental information of each key of each object that needs to be recovered.

[0120] The OSD recovery module 705 is used to send the last incremental information of each key of each object to be recovered to the slave OSD when the master OSD calculates that the slave OSD is missing a key; the slave OSD performs the operation corresponding to the last incremental information of each key of each object to be recovered based on the last incremental information of each key.

[0121] The master OSD recovery module 706 is used to perform the operation corresponding to the last incremental information of each key of each object that needs to be recovered when the master OSD calculates that the master OSD is missing a key.

[0122] The specific implementation details of each module can be found in the above description of the limitations of a method for restoring metadata from a distributed object storage system, and will not be repeated here.

[0123] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored relating to all or part of the processes in the methods of the above embodiments.

[0124] In one embodiment, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement all or part of the processes in the methods of the above embodiments.

[0125] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.< / string> < / string>

Claims

1. A method for recovering metadata from distributed object storage, characterized in that, Applied to cloud service architecture, the method includes: S1 generates incremental logs when performing metadata operations; S2, the main OSD calculates the data missing status of the placement group based on the full log; S3, When performing data recovery of the target index file, the main OSD traverses the incremental log for each object that needs to be recovered, and finds all incremental information for each key of each object that needs to be recovered; S4, the master OSD determines the last incremental information of each key of each key of each object that needs to be recovered from all incremental information; S5, when the master OSD calculates that the slave OSD is missing a key, the master OSD sends the last incremental information of each key of each object that needs to be recovered to the slave OSD; the slave OSD performs the operation corresponding to the last incremental information of each key of each object that needs to be recovered based on the last incremental information of each key. S6. When the master OSD calculates that the master OSD is missing a key, the master OSD performs the operation corresponding to the last incremental information of each key of each object that needs to be recovered, based on the last incremental information of each key of each object that needs to be recovered.

2. The method for recovering distributed object storage metadata according to claim 1, characterized in that, The generation of incremental logs includes: Receive metadata operation requests initiated by the client; After the placement group processes the metadata operation request, it records the incremental information corresponding to the metadata operation request in the log to form an incremental log. Write the incremental log to disk.

3. The method for recovering distributed object storage metadata according to claim 1, characterized in that, The incremental information includes the operation type, the name of the key, and the version number.

4. The method for recovering distributed object storage metadata according to claim 1, characterized in that, Step S5 includes: The master OSD packages the operations corresponding to the last incremental information of each key of each object that needs to be recovered into a data structure based on the last incremental information of each key of each object that needs to be recovered, and sends the data structure to the slave OSD. The data structure is parsed from the OSD, and corresponding operations are performed on each key of each object that needs to be recovered based on the parsing results.

5. The method for recovering distributed object storage metadata according to claim 4, characterized in that, The data structure includes a mapping structure omap_entries for performing set operations and a set keys_to_remove for performing remove operations.

6. The method for recovering distributed object storage metadata according to claim 1, characterized in that, The method further includes: When performing data recovery on the target index file, the metadata is written to a temporary index file; After data recovery is complete, suspend IO and merge the target index file and temporary index file.

7. A system for recovering metadata from distributed object storage, characterized in that, The system, configured within a cloud service architecture, includes: The incremental log generation module is used to generate incremental logs when performing metadata operations; The data missing situation calculation module is used by the main OSD to calculate the data missing situation of the placement group based on the full log; The incremental log traversal module is used to, when performing data recovery of the target index file, allow the main OSD to traverse the incremental log for each object that needs to be recovered, and find all incremental information for each key of each object that needs to be recovered. The incremental information determination module is used by the main OSD to determine the last incremental information of each key of each object that needs to be recovered from all the incremental information of each key of each object that needs to be recovered; The OSD recovery module is used when the master OSD calculates that the slave OSD is missing a key. The master OSD sends the last incremental information of each key of each object that needs to be recovered to the slave OSD. The slave OSD then performs the corresponding operation for each key of each object that needs to be recovered based on the last incremental information of each key. The master OSD recovery module is used when the master OSD calculates that a key is missing. The master OSD then performs the operation corresponding to the last incremental information of each key of each object that needs to be recovered, based on the last incremental information of each key.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 6.

Images