A lock-free data recycling method based on SSTable

Abstract

The present application relates to a kind of based on SSTable lockless data recycling method, belong to cloud backup field, comprising: the fingerprint SSTable file for the different fingerprint information fp state of characterizing data slice is configured;The reference count of fp is counted;Load fp_alive table and fp_forget table into memory;The fp of reference count 0 is transferred from fp_alive table to fp_forget table;Check consistency time point;Scan fp_forget table, check the count of fingerprint information fp, if it is non 0, then fingerprint information fp is rolled back from fp_forget table to fp_alive table;If the count of fp_forget table is 0, then fp is transferred from fp_forget table and deleted to fp_delete file.The method of the present application realizes data recycling concurrent with backup business, reduces the influence of data recycling on backup system business.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of cloud disaster recovery and data recycling, and particularly relates to a lock-free data recycling method based on SSTable. BACKGROUND

[0002] With the rapid development of cloud computing, big data and the Internet of Things, the data on the Internet is growing explosively. These data not only occupy a huge storage space, but also a large amount of business data is related to the lifeblood of enterprises. Therefore, in order to prevent business interruption caused by external factors and bring losses to customers, large enterprises will purchase a large number of disaster recovery systems to ensure business continuity. Since the disaster recovery system will backup a large amount of production data every day, but considering the backend storage cost, the customer will generally only keep the backup data of the latest few versions, so the disaster recovery system needs to recycle expired data while performing daily backup. According to the type and number of files backed up by the user, the object of data recycling may be a single large file or a large number of small files. This brings challenges to the read-write performance of the disaster recovery system.

[0003] The traditional disaster recovery software will face the following problems when recycling data:

[0004] 1. Users in the disaster recovery system will create periodic backup tasks, and these backup tasks will also set a retention time, that is, the snapshots generated by the backup will automatically expire, and the data recycling and periodic backup tasks need to be handled at the same time. Especially in the scenario of periodic backup of a large number of small files, the amount of data to be backed up is very large, and the amount of automatically expired data is also very large. How to ensure that the disaster recovery system can normally backup and restore performance without being affected when a large amount of data is recycled is a common problem.

[0005] 2. The disaster recovery system generally has a normal backup mode and a deduplication mode. In the deduplication mode, the data is sliced and the fingerprints of the slices are saved to the database. When subsequent backup is performed, the corresponding fingerprint information is queried to reduce the data transmission of the corresponding slice, thereby reducing the backend storage space occupation and improving the transmission speed. The data recycling task will reduce the reference count of the fingerprint, and only when the reference count of the fingerprint is reduced to 0, the corresponding slice can be recycled. When the data recycling task loads the fingerprint information fp and calculates in the memory that the reference count is 0, there may be concurrent backup business that references this fp and increases the reference count to more than 0 at this time. This leads to inconsistency between the reference count of the fp read by the data recycling task and the actual reference count. If such fp is deleted recklessly, it will cause the backup snapshot to be unavailable, affecting the backup and recovery business.

[0006] To solve the above two problems, a common solution is to recover data by time period and lock mutual exclusion, that is, to initiate a data recovery task and mutual exclusion (task-level mutual exclusion or fingerprint-level mutual exclusion) during a time period when the user traffic is low, for example, no backup / restoration can be initiated when the data recovery task is running, or the data recovery task is not allowed to read and write fingerprint data when the backup / restoration task reads and writes fingerprint data, so as to ensure the consistency of the reference count and avoid the problem of accidental deletion.

[0007] However, the recovery by time period and task-level mutual exclusion may cause the periodic task of the user to be in a suspended state at a certain time period, and the fingerprint-level mutual exclusion may reduce the speed of backup and restoration (locking and unlocking are required for each fingerprint operation), and once the data volume becomes large, the overhead of the mutual exclusion lock will seriously affect the performance of the backup and restoration service, which seriously affects the efficient operation of the disaster recovery service, especially for the backup service with a short backup period and a large amount of backup data, causing the periodic task of the user to be in a suspended state or the backup performance to be reduced at a certain time period. Therefore, the traditional solution only alleviates the above problems, but does not completely solve the problems. SUMMARY

[0008] In view of the deficiencies of the prior art, the purpose of the application is to provide a lock-free data recovery method based on SSTable, which can realize concurrent data recovery with a disaster recovery service and reduce the impact of data recovery on the disaster recovery system service.

[0009] The application provides a lock-free data recovery method based on SSTable, comprising:

[0010] S1, configuring a fingerprint SSTable file for representing the state of different fingerprint information fp of a data slice, wherein the fingerprint SSTable file comprises an fp_alive table, an fp_forget table and an fp_delete table;

[0011] S2, counting the reference count of the fingerprint information fp;

[0012] S3, loading the fp_alive table and the fp_forget table into the memory;

[0013] S4, transferring the fingerprint information fp with a reference count of 0 from the fp_alive table to the fp_forget table;

[0014] S5, checking a consistency time point, comprising: after the data recovery thread writes the fingerprint information fp from the fp_alive table to the fp_forget table, waiting for all the ongoing backups to be completed, and taking the time point when the backup is completed as the consistency time point;

[0015] S6. After the consistency time point is reached, scan the fp_forget table and check if the count of the fingerprint information fp is 0. If it is non-zero, it means that it has been backed up and reused. Roll back the above fingerprint information fp from the fp_forget table to the fp_alive table.

[0016] S7. After the consistency time point is reached, scan the fp_forget table and check if the count of the fp_forget table is 0. If it is 0, it means that there will be no backup reference. Move the above fingerprint information fp from the fp_forget table to the fp_delete table.

[0017] S8. Scan the fp_delete table to obtain the deletable fingerprint information fp and the corresponding object. If all the fingerprint information fp in the object exists in the fp_delete table, then delete the object directly.

[0018] Furthermore, in S1,

[0019] The fp_alive table records fp blocks with a reference count > 0;

[0020] The fp_forget table records fp blocks whose reference counts are reduced to 0 after data reclamation due to snapshot expiration;

[0021] The fp_delete table records fp blocks with a reference count of 0 that can be deleted.

[0022] Furthermore, in S1, each fingerprint SSTable file defines a fingerprint information fp block, and the fingerprint SSTable file includes: fingerprint information fp, SSTable metadata and Bloom filter.

[0023] Furthermore, the fingerprint information fp includes: fingerprint hash fphash, the object uuid where the data slice corresponding to fp is located, the offset of the data slice in the object, and the size of the data slice in the object;

[0024] The SSTable metadata includes: the starting fphash and ending fphash of the fp block, the total size of the current fp block, and the number of fp blocks contained within it;

[0025] The Bloom filter is used to accelerate the search for fingerprint information fp, including: first, determining the fp block to which the fingerprint information fp belongs based on the fingerprint hash fphash, and then using the Bloom filter to determine whether the fingerprint information fp is in the corresponding fingerprint SSTable file. If it exists, the fingerprint SSTable file is loaded and a binary search is performed within it. If it does not exist, it is determined that the fingerprint information fp file does not exist.

[0026] Furthermore, the step of determining the fp block to which the fingerprint information fp belongs based on the fingerprint hash fphash includes: after loading all SSTable metadata, performing a binary search on all starting fphash and ending fphash to determine the fp block to which it belongs.

[0027] Furthermore, in S2, all fingerprint information fp under the snapshot is read, the count of the fingerprint information fp under the corresponding available snapshot is increased, and the count of the fingerprint information fp under the snapshot marked for deletion is decreased.

[0028] Furthermore, in S4, the fingerprint information fp with a reference count of 0 is written to the fp_forget table. If there is concurrent backup at this time, when data is recycled, the fingerprint information fp with a reference count of 0 will be backed up and referenced, and the count will be increased to >0. There is no mutual exclusion between reading and writing fingerprint information fp, so the data in the fp_forget table can be deleted immediately and wait for subsequent processing.

[0029] Furthermore, in S5, the fingerprint information fp in the fp_forget table is backed up and reused when the following timing conditions exist:

[0030] (1) Data reclamation reads fp_x, whose reference count is 0, from the fp_alive table;

[0031] (2) During backup, the fingerprint information of a certain data fragment is calculated as fp_x;

[0032] (3) During backup, fp_x was found to exist in the fp_alive table;

[0033] (4) Data recycling moves fp_x from the fp_alive table to the fp_forget table;

[0034] (5) Increment the reference count of fp_x by 1 during backup.

[0035] Furthermore, in S5, let the data backup time period be [T0, T2], and the data reclamation time period be [T1, T3], where data reclamation and data backup occur concurrently, T0 <T1<T2;

[0036] When time T2 is reached, all backups have been completed and no new backups have been generated. Time T2 is the consistency point. At this time, the fingerprint information fp before time T2 is no longer mistakenly deleted.

[0037] Furthermore, in S8, if the object contains fingerprint information fp that needs to be retained, the object is downloaded into memory, the fingerprint information fp that needs to be retained is aggregated and re-uploaded, and the offset in the retained fingerprint information fp record is updated after the upload is completed.

[0038] The beneficial effects of this invention are as follows:

[0039] (1) The traditional disaster recovery software data recycling is optimized so that the new data recycling method can use SSTable to accelerate the search process, especially for scenarios with a large number of small files, which can improve the data recycling speed by more than 20%.

[0040] (2) The traditional disaster recovery software data reclamation is optimized so that the new data reclamation method can achieve lock-free data reclamation by flowing between three types of SSTables, avoiding the overhead of locking and mutual exclusion, and greatly improving the speed of data reclamation through the lock-free design. While ensuring the consistency of fingerprint information, the impact of data reclamation on the performance of backup and recovery services is reduced.

[0041] (3) The traditional disaster recovery software data reclamation is optimized so that the new data reclamation method can detect consistency time points, reduce the probability of accidental deletion, ensure fingerprint read and write consistency, and avoid snapshot unavailability caused by fingerprint inconsistency. The data reclamation method proposed in this invention can reduce the probability of accidental deletion to 0. Attached Figure Description

[0042] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0043] Figure 1 This is a flowchart of a lock-free data reclamation method based on SSTable according to an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of a fingerprint SSTable file according to an embodiment of the present invention;

[0045] Figure 3 This is a flowchart illustrating the FP recycling process according to an embodiment of the present invention;

[0046] Figure 4 This is a timing diagram of the fp transfer in an embodiment of the present invention. Detailed Implementation

[0047] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0048] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0049] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0050] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0051] The technical terms involved in this invention are explained below:

[0052] SSTable (sorted string table): An internally sorted and persistent key / value pair storage method. Each SSTable file stores a certain range of key-value pairs. The ranges of key-value pairs between SSTable files do not overlap and are ordered. When searching, the range of key-value pairs is first located based on the key, then the corresponding SSTable file is loaded into memory, and a binary search is performed. When writing, the key-value pairs in memory are sorted and then flushed to disk. The advantage is that key-value pairs can be searched concurrently and the query speed is fast.

[0053] Fp (fingerprint): After slicing the data, a feature value is calculated for each slice using a specific algorithm. This feature value represents the unique characteristics of the slice. If two slices have the same fingerprint, the data content of the two slices is considered to be completely consistent. The selected algorithm ensures that the probability of hash collision is minimized.

[0054] To address the challenge of periodically reclaiming massive amounts of small files, this patent proposes a lock-free data reclamation method based on SSTable, which ensures the consistency of fingerprint information while improving the efficiency of data reclamation.

[0055] This invention proposes a lock-free data reclamation method based on SSTable, such as... Figure 1 As shown, it includes the following steps:

[0056] S1, Configure the fingerprint SSTable file to represent the different fingerprint information fp states of the data slice.

[0057] This invention introduces three types of SSTable files to control the state of fp. For example... Figure 2 As shown, the three fingerprint SSTable files are: fp_alive, fp_forget, and fp_delete.

[0058] The fp_alive table records fp blocks with a reference count greater than 0.

[0059] The fp_forget table records fp blocks whose reference counts were reduced to 0 after data reclamation due to snapshot expiration.

[0060] The fp_delete table records fp blocks with a reference count of 0 that can be deleted.

[0061] Each fingerprint SSTable file defines a fingerprint information fp block. The basic structure of each SSTable includes: fingerprint information fp (fp record itself), SSTable metadata (meta block), and Bloom filter.

[0062] The fingerprint information fp includes: fingerprint hash fphash, the UUID of the object where the data slice corresponding to fp is located, the offset of the data slice in the object, and the size of the data slice in the object.

[0063] A fingerprint SSTable file defines an fp block. The SSTable metadata includes: the start fphash and end fphash of the fp block, the total size of the current fp block, and the number of fp contained within it, etc.

[0064] Bloom filters are used to speed up the search for fingerprint information (fp).

[0065] First, the fingerprint hash fphash is used to determine the FP block to which the fingerprint information fp belongs. Specifically, after loading all SSTable metadata, a binary search is performed on all start and end fphashes to determine the corresponding FP block.

[0066] Then, a Bloom filter is used to determine whether the fingerprint information fp exists in the corresponding fingerprint SSTable file. If it exists, the fingerprint SSTable file is loaded and a binary search is performed within it. If it does not exist, it is determined that the fingerprint information fp file does not exist.

[0067] As disaster recovery operations continue, a large number of backup snapshots gradually expire, and the system begins data reclamation. The reclamation process is described in the following reference. Figure 3 As shown.

[0068] S2, count the reference count of fingerprint information fp.

[0069] Read the fingerprint information fp under all snapshots, increment the count of the fingerprint information fp under the corresponding available snapshot, and decrement the count of the fingerprint information fp under the snapshot marked for deletion.

[0070] S3, load the fp_alive and fp_forget tables into memory.

[0071] Specifically, the fingerprint information fp is loaded, and the fingerprint SSTable: fp_alive table and fp_forget table are loaded into memory to facilitate the subsequent distribution of these fingerprint information fp according to reference counting.

[0072] S4, transfer the fingerprint information fp with a reference count of 0 from the fp_alive table to the fp_forget table.

[0073] Specifically, the fingerprint information fp with a reference count of 0 is written into the fp_forget table. Since there may be concurrent backups at this time, when the data is recycled, the fingerprint information fp with a count of 0 read may be re-referenced by the backup and the count increased to >0. That is, there is no locking and mutual exclusion between reading and writing the fingerprint information fp. Therefore, the data in the fp_forget table cannot be immediately deleted and needs to wait for subsequent processing.

[0074] S5, Check the consistency time point, including: after the data recycling thread finishes writing the fingerprint information fp from the fp_alive table into the fp_forget table, wait for all ongoing backups to complete, and use the backup completion time as the consistency time point.

[0075] Specifically, at the time point when the fingerprint SSTable:fp_forget table writing is completed, there may still be ongoing backups, and these backups may re-reference the fingerprint information fp in the fp_forget table. Therefore, after the data recycling thread finishes writing into the fp_forget table, it needs to wait for all ongoing backups to complete, and the completion time is the consistency time point.

[0076] Figure 4 It represents the fp transfer timing diagram for data recycling and data backup concurrency. In this timing diagram, the fingerprint information fp generated by the backup snap1 before time T0 is recorded in the fp_alive table. The data backup snap2 starts at time T0 and ends at time T2. The data recycling starts at time T1 (T0 < T1 < T2) and ends at time T3 (T1 < T2 < T3).

[0077] There is concurrency between data recycling and data backup: when performing the snap2 backup, it checks whether the data shards are repeated. If they are repeated, the data shard does not need to be transferred. Therefore, when performing the snap2 backup, it needs to query the fp_alive table to determine whether the fp corresponding to the data shard of snap2 already exists. If it exists, the reference count of the corresponding fp is increased. If it does not exist, a new fp is inserted into the fp_alive table.

[0078] Meanwhile, if the data recycling is triggered at T1 (T0 < T1 < T2), the data recycling needs to check whether the reference count of the fp in fp_alive is 0. If it is 0, it will be recycled. The fingerprint information fp is transferred from the fp_alive table to the fp_forget table.

[0079] When there is the following timing sequence from (1) to (5), the fp in the fp_forget table will be re-referenced by the backup:

[0080] (1) The data recycling reads the fp_x with a reference count of 0 in fp_alive;

[0081] (2) During snap2 backup, the fingerprint of a certain data fragment is calculated as fp_x;

[0082] (3) During snap2 backup, fp_x was found to exist in fp_alive;

[0083] (4) Data recycling moves fp_x from fp_alive to fp_forget;

[0084] (5) When performing a snap2 backup, increment the reference count of fp_x by 1.

[0085] Where fp_x is the fp in the fp_alive table with a reference count of 0.

[0086] From the perspective of data reclamation, the reference count of fp_x transferred to the fp_forget table is 0 (it was read as 0 before being transferred). However, the actual snap2 backup re-references this fp_x and increments its count by 1. If fp_x is directly deleted from the fp_forget table at this point, accidental deletion will occur.

[0087] Therefore, data recycling needs to detect whether there are any backups currently in progress. Figure 4 The system executes snapshot 2 and waits for these backups to complete. A consistency point in time is reached when all backups in progress have completed and no new backups have been generated.

[0088] The data reclamation process assumes that the file descriptors (fp) in the current fp_forget table can only be affected by snapshots before the consistency point T2. This means it needs to check if the reference count in the fp_forget table is 0 and move the fp to the fp_delete table or the fp_alive table. This transfer process (from time T1 to time T3) is different from the backup process after the consistency point T2. Figure 4 The shift process is unrelated to the snapshot 3 in the context of consistency, meaning that the shift process can run concurrently with backups after the consistency point T2.

[0089] The reasoning is as follows: If snap3 backs up fingerprint information (fp) that overlaps with those from time points T2 (snap2, snap1, etc.), these fp will exist in the current fp_alive and fp_forget tables (the fp_forget table will be processed). If snap3 backs up new fp, they can be directly inserted into the fp_alive table. After this transfer process is complete, data reclamation will generate a new fp_forget table and wait for the next consistency point in time.

[0090] In summary, the time period for data backup snap2 is [T0, T2]; the time period for data recovery is [T1, T3]. Since T0 < T1 < T2, that is, there is concurrency between data recovery and data backup, the reference count of fingerprints may be inconsistent during the time period [T1, T2]. Only when reaching the moment of T2, all backups are completed and no new backups are generated, that is, the moment of T2 is the consistency time point. At this time, there will be no problem of incorrect deletion of the fingerprint information fp before the moment of T2 during data recovery.

[0091] In summary, in the present invention, there is no need for task-level mutual exclusion or fingerprint-level mutual exclusion between data recovery and data backup. The fingerprint information fp ensures data consistency by transferring among three types of fingerprint SSTables, improving the efficiency of concurrent execution with backup / restore.

[0092] S6, after reaching the consistency time point, scan the fp_forget table to check whether the count of the fingerprint information fp is 0. If it is non-0, it means that it has been re-referenced by the backup. Roll back the above fingerprint information fp from the fp_forget table to the fp_alive table.

[0093] Specifically, transfer the fp_forget table to the fp_alive table. After reaching the consistency time point, scan the fingerprint SSTable: fp_forget to check whether the count of fp is 0. If it is non-0, it means that it has been re-referenced by the backup, and these fingerprint information fp need to be rolled back to the fingerprint SSTable: fp_alive table.

[0094] S7, after reaching the consistency time point, scan the fp_forget table to check whether the count of the fp_forget table is 0. If it is 0, it means that there will be no backup reference. Transfer and delete the above fingerprint information fp from the fp_forget table to the fp_delete table.

[0095] Specifically, transfer the fp_forget table to the fp_delete table. After reaching the consistency time point, scan the fingerprint SSTable: fp_forget to check whether the count of the fingerprint information fp is 0. If it is 0, it means that there must be no backup reference, and these fingerprint information fp can be deleted to the fingerprint SSTable: fp_delete table.

[0096] S8, scan the fp_delete table to obtain the deletable fingerprint information fp and the corresponding object. When all the fingerprint information fp in the object exists in the fp_delete table, directly delete the object.

[0097] Specifically, scan the fingerprint SSTable: fp_delete table to obtain the deletable fp and the corresponding object.

[0098] If all fingerprint information fp in the object is in the fp_delete table, then the object is deleted directly.

[0099] If the object contains fingerprint information (fp) that needs to be retained (not belonging to the fp_delete table), then download the object to memory, aggregate the fingerprint information fp that needs to be retained, and re-upload it (the path remains the same). After the upload is complete, update the offset in the retained fingerprint information fp record. After re-aggregation, the object UUID remains the same, but the offset changes, requiring an update.

[0100] The lock-free data reclamation method based on SSTable of the present invention designs three types of SSTable (fp_alive table, fp_forget table and fp_delete table). During the data reclamation process, fp flows between these three types of SSTable. Fingerprint read and write consistency is guaranteed by judging the consistency time point. At the same time, the lock-free design has higher data reclamation efficiency in the scenario of massive small files and will not affect backup and recovery.

[0101] In summary, the lock-free data reclamation method based on SSTable proposed in this invention enables concurrent data reclamation with disaster recovery operations. It divides fingerprint file (FP) records into three types: SSTable: fp_alive / fp_forget / fp_delete. When disaster recovery and data reclamation occur concurrently, the fingerprint FP flows between these three SSTables. This ensures fingerprint FP consistency without requiring locking and mutual exclusion between data reclamation and backup / recovery tasks, and improves data reclamation efficiency. Especially in scenarios involving the periodic backup / deletion of massive amounts of small files, it can reduce the impact of data reclamation on disaster recovery system operations.

[0102] The lock-free data reclamation method based on SSTable of the present invention can be applied to the following scenarios:

[0103] 1. China Telecom Cloud Elastic Storage Product Line: Cloud-based and on-premises disaster recovery and backup services.

[0104] 2. Any disaster recovery product with deduplication functionality.

[0105] 3. Optimized specifically for scenarios involving the periodic backup and deletion of massive amounts of small files.

[0106] The lock-free data reclamation method based on SSTable of the present invention has the following beneficial effects:

[0107] (1) The traditional disaster recovery software data recycling is optimized so that the new data recycling method can use SSTable to accelerate the search process, especially for scenarios with a large number of small files, which can improve the data recycling speed by more than 20%.

[0108] (2) The traditional disaster recovery software data reclamation is optimized so that the new data reclamation method can achieve lock-free data reclamation by flowing between three types of SSTables, avoiding the overhead of locking and mutual exclusion, and greatly improving the speed of data reclamation through the lock-free design. While ensuring the consistency of fingerprint information, the impact of data reclamation on the performance of backup and recovery services is reduced.

[0109] (3) The traditional disaster recovery software data reclamation is optimized so that the new data reclamation method can detect consistency time points, reduce the probability of accidental deletion, ensure fingerprint read and write consistency, and avoid snapshot unavailability caused by fingerprint inconsistency. The data reclamation method proposed in this invention can reduce the probability of accidental deletion to 0.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A lock-free data reclamation method based on SSTable, characterized in that, include: S1, Configure a fingerprint SSTable file for characterizing different fingerprint information fp states of data slices, the fingerprint SSTable file includes: fp_alive table, fp_forget table and fp_delete table; S2, count the reference count of fingerprint information fp; S3, load the fp_alive and fp_forget tables into memory; S4, transfer the fingerprint information fp with a reference count of 0 from the fp_alive table to the fp_forget table; S5, check the consistency time point, including: after the data recycling thread finishes writing the fingerprint information fp from the fp_alive table to the fp_forget table, it waits for all ongoing backups to complete, and takes the backup completion time as the consistency time point; S6. After the consistency time point is reached, scan the fp_forget table and check if the count of the fingerprint information fp is 0. If it is non-zero, it means that it has been backed up and reused. Roll back the above fingerprint information fp from the fp_forget table to the fp_alive table. S7. After the consistency time point is reached, scan the fp_forget table and check if the count of the fp_forget table is 0. If it is 0, it means that there will be no backup reference. Move the above fingerprint information fp from the fp_forget table to the fp_delete table. S8. Scan the fp_delete table to obtain the deletable fingerprint information fp and the corresponding object. If all the fingerprint information fp in the object exists in the fp_delete table, then delete the object directly.

2. The lock-free data reclamation method based on SSTable according to claim 1, characterized in that, In S1, The fp_alive table records fp blocks with a reference count > 0; The fp_forget table records fp blocks whose reference counts are reduced to 0 after data reclamation due to snapshot expiration; The fp_delete table records fp blocks with a reference count of 0 that can be deleted.

3. The lock-free data reclamation method based on SSTable according to claim 1, characterized in that, In S1, each fingerprint SSTable file defines a fingerprint information fp block, and the fingerprint SSTable file includes: fingerprint information fp, SSTable metadata and Bloom filter.

4. The lock-free data reclamation method based on SSTable according to claim 3, characterized in that, The fingerprint information fp includes: fingerprint hash fphash, the object uuid where the data slice corresponding to fp is located, the offset of the data slice in the object, and the size of the data slice in the object; The SSTable metadata includes: the starting fphash and ending fphash of the fp block, the total size of the current fp block, and the number of fp blocks contained within it; The Bloom filter is used to accelerate the search for fingerprint information fp, including: first, determining the fp block to which the fingerprint information fp belongs based on the fingerprint hash fphash, and then using the Bloom filter to determine whether the fingerprint information fp is in the corresponding fingerprint SSTable file. If it exists, the fingerprint SSTable file is loaded and a binary search is performed within it. If it does not exist, it is determined that the fingerprint information fp file does not exist.

5. The lock-free data reclamation method based on SSTable according to claim 4, characterized in that, The step of determining the FP block to which the fingerprint information fp belongs based on the fingerprint hash fphash includes: after loading all SSTable metadata, performing a binary search on all start fphash and end fphash to determine the FP block to which it belongs.

6. The lock-free data reclamation method based on SSTable according to claim 1, characterized in that, In S2, all fingerprint information fp under the snapshot is read, the count of the fingerprint information fp under the corresponding available snapshot is increased, and the count of the fingerprint information fp under the snapshot marked for deletion is decreased.

7. The lock-free data reclamation method based on SSTable according to claim 1, characterized in that, In S4, The fingerprint information fp with a reference count of 0 is written to the fp_forget table. If there is concurrent backup at this time, when the data is recycled, the fingerprint information fp with a reference count of 0 will be backed up and referenced, and the count will be increased to >0. There is no mutual exclusion between reading and writing fingerprint information fp, so the data in the fp_forget table can be deleted immediately and wait for subsequent processing.

8. The lock-free data reclamation method based on SSTable according to claim 1, characterized in that, In S5, when the following timing sequence from (1) to (5) exists, the fingerprint information fp in the fp_forget table is backed up and reused: (1) Data reclamation reads fp_x, whose reference count is 0, from the fp_alive table; (2) During backup, the fingerprint information of a certain data fragment is calculated as fp_x; (3) During backup, fp_x was found to exist in the fp_alive table; (4) Data recycling moves fp_x from the fp_alive table to the fp_forget table; (5) Increment the reference count of fp_x by 1 during backup.

9. The lock-free data reclamation method based on SSTable according to claim 1, characterized in that, In S5, the data backup time period is [T0, T2], and the data reclamation time period is [T1, T3]. Data reclamation and data backup occur concurrently. <T1<T2; When time T2 is reached, all backups have been completed and no new backups have been generated. Time T2 is the consistency point. At this time, the fingerprint information fp before time T2 is no longer mistakenly deleted.

10. A lock-free data reclamation method based on SSTable according to claim 1, characterized in that, In S8, if the object contains fingerprint information fp that needs to be retained, the object is downloaded to memory, the fingerprint information fp that needs to be retained is aggregated and then re-uploaded. After the upload is completed, the offset in the retained fingerprint information fp record is updated.

Images