Distributed object storage metadata recovery method and system and storage medium

Abstract

The invention discloses a distributed object storage metadata recovery method and system and a storage medium. The method comprises the steps that when metadata operation is carried out, an incremental log is generated; the main OSD traverses the increment log for each object needing to be recovered, and all increment information of each key of each object needing to be recovered is found; the main OSD determines the last piece of increment information of each key of each object needing to be recovered in all increment information of each key of each object needing to be recovered; when the OSD lacks the key, executing an operation corresponding to the last piece of increment information on each key of each object needing to be recovered; when the main OSD lacks the key, the main OSD executes the operation corresponding to the last piece of increment information on each key of each object needing to be recovered. According to the method and the device, incremental recovery can be adopted during data recovery, so that all key value pairs are prevented from being traversed, the data recovery time is shortened, and the recovery speed and efficiency of the metadata are greatly improved.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a method, system and storage medium for recovering distributed object storage metadata. Background Art

[0002] In distributed storage systems, metadata management and storage are important links to ensure efficient read and write performance. In recent years, with the rapid development of big data, the LSM-Tree (Log-Structured Merge-Tree) structure has been widely used due to its efficient write performance. There are many systems that store metadata based on LSM tree technology. The core idea of ​​LSM-Tree is to use sequential writes to improve write performance; LSM-Tree is not a tree data structure, but only a storage structure. A typical example is rocsdb, which also uses LSM TREE technology to store system metadata and gradually pushes the metadata to different levels. The written data will first be placed in the L0 layer. When each layer reaches a certain threshold, the data will be transferred to a deeper layer, that is, L0->L1, L1->L2, and so on. At the same time, each layer has its own fixed storage file (generally multiple files are configured for each layer) to store these key-value pairs.

[0003] For distributed storage systems, storing data metadata in rocsdb is a common choice. In order to ensure data reliability, metadata is usually stored in multiple copies, and usually three copies of metadata are stored in the system. Since the data for a cluster is 1 billion or even tens of billions, a concept of placement group PG (placement group) is designed. All written data will calculate a hash value and disperse it into each placement group. The number of placement groups is generally controlled within thousands, which can avoid the hot spot effect of writing and can also perform data management well.

[0004] At the same time, each placement group is usually composed of object storage devices (OSDs) on different nodes. Usually, each OSD corresponds to a disk, and each OSD runs an independent rocksdb. Figure 1 As shown in the figure, after receiving the request, the master osd will write the metadata to the corresponding rocsdb, and push the data to other slave osds in the same PG, which can effectively ensure the reliability of the data.

[0005] However, when one of the PG copies is lost or inconsistent with the other copies, the system's data recovery process will be triggered. The data recovery process compares the logs recorded by each osd, calculates the data set that is missing from the other placement group, and copies each data from the complete metadata node to the rocksdb on the missing node in turn according to this set.

[0006] The current metadata information is recorded in rocksdb. For the convenience of management, when operating a directory or a bucket, a unique and fixed prefix related to the directory or bucket name will be used as the storage key, and the value is the metadata to be stored.

[0007] Taking object storage as an example, the metadata of each object in a bucket is distributed on the index shards of multiple buckets according to the hash. Each index shard file will store thousands of key-value pairs, and the prefixes of these key-value pairs are the same.

[0008] To ensure the atomicity of IO, the metadata on the same index shard is queued in PG. The scheduling is based on the index shard as the smallest unit. Figure 2 As shown in the figure, when the index shard is recovering data, all metadata requests on the same index shard need to wait until the index shard is recovered before writing the metadata. If the index shard is recovered too slowly, the IO will be blocked and even a large number of client requests will fail.

[0009] The recovery speed of each index shard is positively correlated with the key-value pairs (meta information) associated with each shard, because the data of the index shard is stored in rocksdb, and the data of the index shard is not concentrated in a storage file (SST file) of a certain level or a certain layer of rocksdb, but is scattered in each level and the sst files in each level, so we cannot achieve the effect of restoring the index shard by directly copying the SST file. Instead, we must traverse all the key-value pairs of the complete index shard, and then write all the data to the missing index shard.

[0010] Figure 3 Shows the process of main osd recovering data, from Figure 3 It can be seen that when restoring the index file, it is necessary to traverse all the keys of the existing index objects, then delete them all, and then rewrite the keys read from other osds.

[0011] Assume that an index file has 200,000 keys. When a node loses power, a degraded write occurs in a certain PG, for example, only 2 copies of data are written out of 3 copies. When the osd in this PG rejoins, even if only one metadata (a key) is missing, it must follow the above process again. The local osd traverses and reads 200,000 keys, then deletes them all, reads 200,000 keys from the peer, and then rewrites them locally, resulting in very low recovery efficiency. More seriously, in this process, it is easy to cause too deep iteration of rocksdb IO, which leads to the osd process timeout and crash. Summary of the invention

[0012] The present application provides a method, system and storage medium for recovering metadata of a distributed object storage, aiming to solve the technical problem that the existing data recovery process needs to traverse all keys of existing index objects and the recovery efficiency is low.

[0013] In a first aspect, a method for recovering metadata of a distributed object storage is applied to a cloud service architecture, and the method comprises:

[0014] S1, when performing metadata operations, incremental logs are generated;

[0015] S2: The primary OSD calculates the data missing of the placement group based on the full log.

[0016] S3, when recovering the data of the target index file, the primary OSD traverses the incremental log for each object to be recovered, and finds all incremental information of each key of each object to be recovered;

[0017] S4, the primary OSD determines the last incremental information of each key of each object that needs to be restored from all incremental information of each key of each object that needs to be restored;

[0018] S5, when the master OSD calculates that the slave OSD is missing a key, the master OSD sends the last incremental information of each key of each object that needs to be restored to the slave OSD; the slave OSD performs the operation corresponding to the last incremental information on each key of each object that needs to be restored according to the last incremental information of each key of each object that needs to be restored;

[0019] S6, when the primary OSD calculates that the primary OSD is missing a key, the primary OSD performs the operation corresponding to the last incremental information for each key of each object that needs to be restored based on the last incremental information of each key of each object that needs to be restored.

[0020] In the above solution, optionally, the generation of the incremental log includes:

[0021] Receive metadata operation requests initiated by the client;

[0022] After the placement group processes the metadata operation request, the incremental information corresponding to the metadata operation request is recorded in a log to form an incremental log;

[0023] The incremental log is written to disk.

[0024] In the above solution, optionally, the incremental information includes the operation type, the key name and the version number.

[0025] In the above solution, optionally, step S5 includes:

[0026] The master OSD packages the operations corresponding to the last incremental information of each key of each object to be restored into a data structure according to the last incremental information of each key of each object to be restored, and sends the data structure to the slave OSD;

[0027] The data structure is parsed from the OSD, and corresponding operations are performed on each key of each object that needs to be restored according to the parsing result.

[0028] In the above solution, further optionally, the data structure includes a mapping structure omap_entries for performing a set operation, and a set keys_to_remove for performing a remove operation.

[0029] In the above solution, optionally, the method further includes:

[0030] When recovering the target index file, the metadata is written into a temporary index file;

[0031] After the data recovery is completed, IO is suspended and the target index file and the temporary index file are merged.

[0032] In a second aspect, a distributed object storage metadata recovery system is provided in a cloud service architecture, the system comprising:

[0033] The incremental log generation module is used to generate incremental logs when performing metadata operations;

[0034] The data missing situation calculation module is used by the master OSD to calculate the data missing situation of the placement group based on the full log;

[0035] The incremental log traversal module is used to traverse the incremental log for each object to be restored when restoring the data of the target index file, and find out all the incremental information of each key of each object to be restored;

[0036] An incremental information determination module is used for the primary OSD to determine the last incremental information of each key of each object that needs to be restored from all incremental information of each key of each object that needs to be restored;

[0037] The slave OSD recovery module is used for sending the last incremental information of each key of each object to be recovered to the slave OSD when the master OSD calculates that the slave OSD lacks a key; the slave OSD performs the operation corresponding to the last incremental information on each key of each object to be recovered according to the last incremental information of each key of each object to be recovered;

[0038] The main OSD recovery module is used to execute the operation corresponding to the last incremental information for each key of each object that needs to be recovered based on the last incremental information of each key of each object that needs to be recovered when the main OSD calculates that the main OSD is missing a key.

[0039] According to a third aspect, a computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0040] In a fourth aspect, a computer program product comprises a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0041] Compared with the prior art, this application has at least the following beneficial effects:

[0042] In the method provided in the embodiment of the present application, an incremental log is introduced to track the change operations (such as writing and deleting) of each key by recording the incremental log when the metadata is written and deleted, and then the log is replayed during data recovery to determine the latest status of each key, so that incremental recovery can be used when recovering the index file, and only the missing key-value values ​​need to be paid attention to, thereby avoiding the full traversal of all key-value pairs and a large number of repeated IO operations, reducing the data recovery time and greatly improving the recovery speed and efficiency of the metadata; in addition, in the data recovery scenario, IO is not prone to timeout interruptions, reducing a large amount of IO overhead for recovering data; it also solves the problem of IO blocking during index shard recovery and even client business interruption / failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of osd data operation;

[0044] Figure 2 This is a schematic diagram of the index shard data recovery process;

[0045] Figure 3 This is a schematic diagram of the main osd data recovery process;

[0046] Figure 4 A schematic diagram of a process of recovering a distributed object storage metadata provided by an embodiment of the present application;

[0047] Figure 5 This is a schematic diagram of a method for recording incremental logs in one embodiment of the present application;

[0048] Figure 6 This is a schematic diagram of an incremental log in an embodiment of the present application;

[0049] Figure 7 A block diagram of the module architecture of a distributed object storage metadata recovery system provided for one embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] In the description of the present application: unless otherwise specified, expressions such as “include”, “comprising”, “having”, etc. also mean “not limited to” (certain units, components, materials, steps, etc.).

[0052] In one embodiment, Figure 4 As shown, a method for recovering metadata of a distributed object storage is provided, and the method is applied to a cloud service architecture, and the method includes:

[0053] S1, when performing metadata operations, incremental logs are generated;

[0054] S2: The primary OSD calculates the data missing of the placement group based on the full log.

[0055] S3: When restoring the target index file, the primary OSD traverses the incremental log for each object to be restored and finds all incremental information for each key of each object to be restored.

[0056] S4, the primary OSD determines the last incremental information of each key of each object that needs to be restored from all incremental information of each key of each object that needs to be restored;

[0057] S5, when the master OSD calculates that the slave OSD is missing a key, the master OSD sends the last incremental information of each key of each object that needs to be restored to the slave OSD; the slave OSD performs the operation corresponding to the last incremental information on each key of each object that needs to be restored according to the last incremental information of each key of each object that needs to be restored;

[0058] S6, when the primary OSD calculates that the primary OSD is missing a key, the primary OSD performs the operation corresponding to the last incremental information for each key of each object that needs to be restored based on the last incremental information of each key of each object that needs to be restored.

[0059] Furthermore, if Figure 5 As shown, the recording methods of the incremental log pglog include:

[0060] Receive metadata operation requests initiated by the client;

[0061] After the placement group processes the metadata operation request, it records the incremental information corresponding to the metadata operation request into the log to form an incremental log;

[0062] Write incremental logs to disk.

[0063] The incremental information includes the operation type, key name and version number. The operation type here may be deletion or writing. The final result will be recorded in pglog. For details about how to form pglog, see Figure 6 .

[0064] Further, step S5 includes:

[0065] The master OSD packages the operations corresponding to the last incremental information of each key of each object that needs to be restored into a data structure based on the last incremental information of each key of each object that needs to be restored, and sends the data structure to the slave OSD;

[0066] Parse the data structure from OSD and perform corresponding operations on each key of each object that needs to be restored based on the parsing results.

[0067] The data structure includes a mapping structure omap_entries for performing a set operation and a set keys_to_remove for performing a remove operation.

[0068] Furthermore, the method further comprises:

[0069] When recovering the target index file, the metadata is written into a temporary index file;

[0070] After the data recovery is completed, IO is suspended and the target index file and the temporary index file are merged.

[0071] In the above incremental data recovery solution, the key is how to effectively determine the data items (key-value pairs) that need to be recovered and perform corresponding operations. The following is a detailed explanation of this process:

[0072] After writing the incremental log, how do we determine how many keys need to be restored? When repairing an object (OBJ), the system will obtain the keywords and their corresponding operation records from the incremental log (pglog). The format of the record is as follows:

[0073] {key,{e'v,operation}}

[0074] Each record contains a key, its version number and the operation type (such as set or rm)

[0075] For each object that needs to be repaired, the sender will traverse its related pglog records and search from the back to the front. This is because it hopes to find the last operation on the key to determine the current state. For example:

[0076] Sending side:

[0077] When repairing obj1, the records obtained from pglog are as follows:

[0078] {key0,{10'11,set}{11'14,rm}}

[0079] {key4,{11'15,rm}}

[0080] Then traverse each key, search from back to front, find the first record, if it is rm, the key needs to perform the rm operation, if it is set, the key needs to perform the set operation.

[0081] For key0, the last two records indicate that the first operation is a set operation and the second operation is a rm operation. Since the last record is a rm operation, the rm operation needs to be performed during recovery. For key4, there is only one record, indicating a rm operation. Therefore, the rm operation also needs to be performed during recovery. In this way, the sender can determine which keys need to perform the rm operation during the recovery process.

[0082] For obj1, the data contained in the packaged PushOp is as follows:

[0083] map<string,logentry> omap_entries empty

[0084] set <string>keys_to_remove = {key0,key4}

[0085] After analysis, the sender packages the operations to be performed into a PushOp data structure, including:

[0086] omap_entries: used to store key-value pairs that need to be set. In this example, it is empty because all keys that need to be deleted are rm operations.

[0087] keys_to_remove: stores the keys to be deleted. In this example, keys_to_remove = {key0, key4}.

[0088] The same process is used to repair obj2. After repairing obj2, the records obtained from pglog are as follows:

[0089] {key1,{10'12,rm}{11'16,set}}

[0090] For key1, traverse from back to front and find that the last record is a set operation, so the set operation needs to be performed during the recovery process.

[0091] The data contained in the packaged PushOp is:

[0092] map<string,logentry> omap_entries = {key1, bl} (assuming bl is the value corresponding to key1)

[0093] set <string>keys_to_remove is empty because there is no key to be deleted.

[0094] After receiving PushOp, the receiving end will perform the following operations:

[0095] For the key-value pairs in omap_entries, perform the set operation. For obj1, no set operation is required at this time. For the key in keys_to_remove, perform the rm operation. For obj1, key0 and key4 need to be deleted.

[0096] In other words, the receiving end parses PushOp, performs set operations on the key and value in omap_entries, and performs rm on keys_to_remove. This avoids restoring the entire index object, and only needs to restore a small number of missing keys.

[0097] In short, the metadata recovery process is controlled by the master osd. A placement group generally has three osds, one master osd and two slave osds. The full log is on the master osd, and the master osd can calculate the missing keys of all osds in the placement group (PG) through the full log information.

[0098] Through log calculation, it is found that the master osd itself lacks a key. At this time, the master osd will actively pull the key from the slave osd that has the key, and then the master osd will write the incremental key. Another case is that the slave osd lacks a key. In this case, the master osd pushes the key to the slave osd, and the slave osd writes the incremental key after receiving the information. Therefore, in the solution proposed in this application, the sending end refers to the master osd and the receiving end refers to the slave osd.

[0099] Through this incremental recovery mechanism, the system only operates on the missing parts, thus avoiding the recovery of the entire index object. This method can significantly improve recovery efficiency, reduce I / O overhead, and ensure that the system can quickly recover to a consistent state when a failure occurs.

[0100] In the above method, for the case where the key is missing from the OSD, the sender is responsible for generating and sending data update requests. The sender extracts the key-value pairs that need to be restored from the log (pglog) and determines which operations need to be performed based on these records. The sender reads the log records to determine the latest status of each key (whether a set operation or an rm operation is required). A PushOp is generated based on the log, and the object contains the operation information that needs to be performed (omap_entries and keys_to_remove). This PushOp is sent to the receiving end.

[0101] PushOp is an encapsulation of a series of operations, usually including data change instructions that need to be executed, including setting the value of a key (set) and deleting a key (remove). PushOp consists of omap_entries and keys_to_remove. omap_entries contains a collection of key-value pairs that need to be set, and keys_to_remove contains a list of keys that need to be deleted.

[0102] omap_entries is a mapping structure used to store key-value pairs that need to be set. Each key corresponds to a log entry (logentry), which indicates the state or value that needs to be set during the recovery process. When the receiving end parses PushOp, it traverses this mapping and performs the corresponding set operation to update the state of the target object. keys_to_remove is a collection used to store keys that need to perform rm operations. These keys are extracted from the log and represent items that should be deleted during the recovery process. When the receiving end parses PushOp, it traverses this collection and performs the corresponding rm operation to ensure that the keys that are no longer needed are removed from the target object.

[0103] The receiving end is responsible for receiving the data update request from the sending end and performing the corresponding operation to update the local data storage. The receiving end parses the received PushOp, performs the set operation according to the key-value in omap_entries, and performs the rm operation according to keys_to_remove.

[0104] In summary, the overall idea of ​​the method provided in the embodiment of the present application is to use incremental recovery when recovering data in an index file, and only the missing key-value values ​​need to be recovered.

[0105] In order to achieve the effect of incremental recovery, it is necessary to record incremental logs when metadata is written and deleted, analyze and merge the logs when data is restored, and then replay the logs. According to the list of missing metadata in the logs, push them to the OSDs that need to be restored in sequence.

[0106] With the advent of the era of big data explosion, the demand for distributed storage systems to store massive files is becoming increasingly strong. By using the method provided in the embodiment of the present application, the modified system can achieve the following advantages:

[0107] 1. When a single OSD is restarted or the entire system is restarted, the IOPS performance of the entire system will not drop by more than 20% (before the transformation, the drop could reach 80% in some scenarios);

[0108] 2. In the data recovery scenario, IO is not prone to timeout interruption;

[0109] 3. Reduce a lot of IO overhead for restoring data;

[0110] 4. Object storage stores metadata in a flat format. By optimizing the recovery speed of metadata, it solves the problem of restricting index size and improves the carrying capacity of a single bucket.

[0111] The method provided in the embodiment of the present application adopts a method of incremental recovery by recording logs, which greatly improves the recovery performance of metadata. The efficient log parsing and merging method can ensure the speed of data recovery while ensuring data consistency, and avoid repeated writing and deletion of the same data.

[0112] The method provided in the embodiment of the present application solves the problem of using a full coverage of all key-value pairs in the index sharding process during the recovery process, avoids a large number of repeated IO operations, and reduces data recovery time; solves the problem of IO blocking and even client business interruption / failure during the index sharding recovery process.

[0113] In addition, when data recovery is required in a failure scenario, the metadata can be written into a temporary index first. When data recovery is complete, the IO can be suspended and the two indexes can be merged.

[0114] It should be understood that although Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 4 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0115] In one embodiment, Figure 7 As shown, a distributed object storage metadata recovery system is provided, which is set in a cloud service architecture. The cloud service architecture is provided with a sending end and a receiving end. The system includes the following program modules:

[0116] The incremental log generation module 701 is used to generate incremental logs when performing metadata operations;

[0117] Data missing situation calculation module 702, used for the primary OSD to calculate the data missing situation of the placement group according to the full log;

[0118] The incremental log traversal module 703 is used for traversing the incremental log for each object to be restored when restoring the data of the target index file, and finding all the incremental information of each key of each object to be restored;

[0119] The incremental information determination module 704 is used for the primary OSD to determine the last incremental information of each key of each object that needs to be restored from all the incremental information of each key of each object that needs to be restored;

[0120] The slave OSD recovery module 705 is used for, when the master OSD calculates that the slave OSD lacks a key, the master OSD sends the last incremental information of each key of each object that needs to be recovered to the slave OSD; the slave OSD performs the operation corresponding to the last incremental information on each key of each object that needs to be recovered according to the last incremental information of each key of each object that needs to be recovered;

[0121] The main OSD recovery module 706 is used to perform the operation corresponding to the last incremental information for each key of each object that needs to be recovered when the main OSD calculates that the main OSD lacks a key.

[0122] The specific implementation content of each module can be found in the above definition of a method for recovering metadata of a distributed object storage, which will not be repeated here.

[0123] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which involves all or part of the processes in the above-mentioned embodiment method.

[0124] In one embodiment, a computer program product is also provided, including a computer program / instruction, which implements all or part of the process in the above-mentioned embodiment method when executed by a processor.

[0125] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.< / string> < / string>

Claims

1. A method for recovering metadata of a distributed object storage, characterized in that: Applied to a cloud service architecture, the method comprises: S1, when performing metadata operations, incremental logs are generated; S2: The primary OSD calculates the data missing of the placement group based on the full log. S3, when recovering the data of the target index file, the primary OSD traverses the incremental log for each object to be recovered, and finds all incremental information of each key of each object to be recovered; S4, the primary OSD determines the last incremental information of each key of each object that needs to be restored from all incremental information of each key of each object that needs to be restored; S5, when the master OSD calculates that the slave OSD is missing a key, the master OSD sends the last incremental information of each key of each object that needs to be restored to the slave OSD; the slave OSD performs the operation corresponding to the last incremental information on each key of each object that needs to be restored according to the last incremental information of each key of each object that needs to be restored; S6, when the primary OSD calculates that the primary OSD is missing a key, the primary OSD performs the operation corresponding to the last incremental information for each key of each object that needs to be restored based on the last incremental information of each key of each object that needs to be restored.

2. The method for recovering distributed object storage metadata according to claim 1, characterized in that: The generation of incremental logs includes: Receive metadata operation requests initiated by the client; After the placement group processes the metadata operation request, the incremental information corresponding to the metadata operation request is recorded in a log to form an incremental log; The incremental log is written to disk.

3. The method for recovering distributed object storage metadata according to claim 1, characterized in that: The incremental information includes the operation type, key name and version number.

4. The method for recovering distributed object storage metadata according to claim 1, characterized in that: Step S5 includes: The master OSD packages the operations corresponding to the last incremental information of each key of each object to be restored into a data structure according to the last incremental information of each key of each object to be restored, and sends the data structure to the slave OSD; The data structure is parsed from the OSD, and corresponding operations are performed on each key of each object that needs to be restored according to the parsing result.

5. The method for recovering distributed object storage metadata according to claim 4, characterized in that: The data structure includes a mapping structure omap_entries for executing a set operation and a collection keys_to_remove for executing a remove operation.

6. The method for recovering distributed object storage metadata according to claim 1, characterized in that: The method further comprises: When recovering the target index file, the metadata is written into a temporary index file; After the data recovery is completed, IO is suspended and the target index file and the temporary index file are merged.

7. A distributed object storage metadata recovery system, characterized in that: Set up in the cloud service architecture, the system includes: The incremental log generation module is used to generate incremental logs when performing metadata operations; The data missing situation calculation module is used by the master OSD to calculate the data missing situation of the placement group based on the full log; The incremental log traversal module is used to traverse the incremental log for each object to be restored when restoring the data of the target index file, and find out all the incremental information of each key of each object to be restored; An incremental information determination module is used for the primary OSD to determine the last incremental information of each key of each object that needs to be restored from all incremental information of each key of each object that needs to be restored; The slave OSD recovery module is used for sending the last incremental information of each key of each object to be recovered to the slave OSD when the master OSD calculates that the slave OSD lacks a key; the slave OSD performs the operation corresponding to the last incremental information on each key of each object to be recovered according to the last incremental information of each key of each object to be recovered; The main OSD recovery module is used to execute the operation corresponding to the last incremental information for each key of each object that needs to be recovered based on the last incremental information of each key of each object that needs to be recovered when the main OSD calculates that the main OSD is missing a key.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images