A method and apparatus for spatial recycling in an online incremental reconstruction process

Abstract

The embodiment of the application provides a space recycling method and device in an online incremental reconstruction process, and relates to the technical field of data processing. The method is specifically as follows: determining a target object in which a data block to be deleted is located according to a target index key in a first target copy, and performing a deletion operation on the target index key in the target object; checking the target object after the target index key is deleted, and if there is no other valid index key in the target object, configuring a minimum deletion timestamp for the target object; when a distributed system background traverses each object in the first target copy, obtaining a latest operation timestamp corresponding to the target object; if the minimum deletion timestamp is the latest operation timestamp of the target object, deleting and recycling the space of the target object; and if a write operation timestamp is greater than the minimum deletion timestamp, the target object is retained. The application can improve the overall performance and data security of a distributed storage framework.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, specifically to a method and apparatus for space reclamation during online incremental reconstruction. Background Technology

[0002] With global digitalization, the scale of metadata is rapidly expanding. Currently, distributed systems are often used to expand storage capacity and processing power by adding new storage nodes to adapt to the growing demand for metadata. Therefore, in a distributed storage framework, effective management of metadata is crucial for system performance and data integrity. To facilitate the location and manipulation of metadata blocks, metadata indexes typically use the format object name-object key-object offset-len. This method clearly identifies the location and size of specific data blocks within a stored object, such as object1-key1-offset1-len1.

[0003] Regarding metadata reclamation mechanisms, two main methods are employed: a backend aggregation process and client-side deletion operations. The backend aggregation process periodically checks metadata to determine if data blocks have been overwritten or deleted, thereby reclaiming space from older metadata overwritten by newer data. This approach is suitable for scenarios where data blocks are frequently updated. Client-side deletion operations, on the other hand, involve users initiating deletion commands through the client, marking a deletion timestamp on the key of the data block corresponding to the metadata. The backend aggregation process then uses this key to actually delete the data block and reclaim the space. This significantly reduces the latency of client-side deletion operations, making it particularly suitable for scenarios where users actively delete data blocks.

[0004] However, the existing mechanism has significant flaws in certain scenarios. When deleting a key within an object, the system checks if all keys under that object have been deleted. If all keys have been deleted, the system assumes the object can also be deleted and adds a recent deletion timestamp. However, in online incremental refactoring scenarios, some keys under an object may not yet be in place due to ongoing migration. In this case, the system may incorrectly identify that there are no valid keys under the object and thus add a deletion timestamp to the object. Even after the relevant keys have been migrated, because the write timestamps added when writing to the new copy during the migration process are less than the object's deletion timestamp, the metadata corresponding to the subsequently migrated data blocks is not visible to the outside world, leading to data loss.

[0005] The aforementioned risk of data loss severely impacts the data reliability and stability of distributed storage frameworks in complex scenarios, necessitating a new technical solution to address these issues and improve the overall performance and data security of distributed storage frameworks. Summary of the Invention

[0006] In view of this, embodiments of this application provide a method and apparatus for space reclamation during online incremental reconstruction, which can prevent virus and Trojan samples from being detected and eliminated in scenarios of automated test case writing with multiple collaborators.

[0007] In a first aspect, embodiments of this application provide a method for space reclamation during online incremental reconstruction, including:

[0008] Receive the deletion instruction for the data block to be deleted sent by the client, and obtain the target index key corresponding to the data block to be deleted according to the deletion instruction;

[0009] Obtain a first target replica in the distributed system; the first target replica is a replica in the distributed system that is in an online incremental reconstruction state;

[0010] Based on the target index key, the target object containing the data block to be deleted is determined in the first target replica, and a deletion operation is performed on the target index key in the target object; the target object includes multiple index keys, including the target index key;

[0011] After deleting the target index key, the target object is checked. If there are no other valid index keys in the target object, a minimal deletion timestamp is configured for the target object. The minimal deletion timestamp is the first timestamp of the start of Greenwich Mean Time.

[0012] When the distributed system backend traverses each object in the first target replica, it obtains the latest operation timestamp corresponding to the target object;

[0013] If there exists a minimum deletion timestamp that is the latest operation timestamp of the target object, then the target object will be deleted and its space reclaimed.

[0014] If a write operation timestamp is greater than the minimum deletion timestamp, then the target object is retained.

[0015] As an optional implementation of this application, the step of determining the target object where the data block to be deleted is located in the first target copy based on the target index key, and performing a deletion operation on the target index key in the target object, includes:

[0016] Based on the target index key, the target object containing the data block to be deleted is determined in the first target replica, and a deletion tag is marked on the target index key; the deletion tag is used to instruct the distributed system to delete the data block to be deleted corresponding to the target index key in the background;

[0017] When the distributed system backend traverses the target object in the first target replica, it deletes the data block to be deleted corresponding to the target index key in the target object according to the deletion tag on the target index key.

[0018] As an optional implementation of this application, obtaining the first target copy in the distributed system includes:

[0019] Obtain the status labels corresponding to multiple replicas in the distributed system, and use the replica with the status label being a reconstruction label as the first target replica.

[0020] As an optional implementation of this application, before obtaining the status tags corresponding to multiple replicas in the distributed system and using the replica with the status tag being a reconstruction tag as the first target replica, the method further includes:

[0021] Identify the states of multiple replicas in the distributed system, and mark replicas in the online incremental reconstruction state with reconstruction tags.

[0022] As an optional implementation of this application, after receiving the deletion instruction of the data block to be deleted issued by the client, and obtaining the target index key corresponding to the data block to be deleted according to the deletion instruction, the method further includes:

[0023] Obtain a second target replica in the distributed system; the second target replica is a replica in the distributed system that is in a normal state.

[0024] Based on the target index key, determine the target object where the data block to be deleted is located in the second target replica, and perform a deletion operation on the target index key in the target object;

[0025] After deleting the target index key, the target object is checked. If there are no other valid index keys in the target object, a normal deletion timestamp is configured for the target object. The normal timestamp is the timestamp corresponding to the current time of the distributed system when it is found that there are no other valid data blocks in the target object.

[0026] When the distributed system backend traverses each object in the second target replica, it obtains the latest operation timestamp corresponding to the target object;

[0027] If the latest operation timestamp corresponding to the target object is the normal deletion timestamp, then the target object will be deleted and its space will be reclaimed.

[0028] As an optional implementation of this application, the method further includes:

[0029] When performing a write operation on the target object, the write operation timestamp corresponding to the time of the write operation is recorded, and the write operation timestamp is configured for the target object.

[0030] Secondly, embodiments of this application provide a space reclamation device for online incremental reconstruction processes, comprising:

[0031] The receiving unit is used to receive the deletion instruction of the data block to be deleted sent by the client, and to obtain the target index key corresponding to the data block to be deleted according to the deletion instruction of the data block to be deleted;

[0032] An acquisition unit is used to acquire a first target replica in the distributed system; the first target replica is a replica in the distributed system that is in an online incremental reconstruction state;

[0033] The deletion unit is configured to determine the target object where the data block to be deleted is located in the first target copy based on the target index key, and perform a deletion operation on the target index key in the target object;

[0034] The inspection unit is used to inspect the target object after the target index key is deleted. If there are no other valid index keys in the target object, a minimal deletion timestamp is configured for the target object. The minimal deletion timestamp is the first timestamp of the start of Greenwich Mean Time.

[0035] The traversal unit is used to obtain the latest operation timestamp corresponding to the target object when the distributed system backend traverses each object in the first target replica;

[0036] The processing unit is configured to delete the target object and reclaim space if the smallest deletion timestamp is the latest operation timestamp of the target object.

[0037] The processing unit is further configured to retain the target object if there is a write operation timestamp greater than the minimum deletion timestamp.

[0038] As an optional implementation of this application, the deletion unit is specifically used to determine the target object where the data block to be deleted is located in the first target replica according to the target index key, and to mark the target index key with a deletion tag; the deletion tag is used to instruct the distributed system to delete the data block to be deleted corresponding to the target index key in the background; when the distributed system in the background traverses the target objects in the first target replica, it deletes the data block to be deleted corresponding to the target index key in the target object according to the deletion tag on the target index key.

[0039] As an optional implementation of this application, the acquisition unit is specifically used to acquire the status tags corresponding to multiple replicas in the distributed system, and to use the replica with the status tag being a reconstruction tag as the first target replica.

[0040] As an optional implementation of this application, the acquisition unit is further configured to identify the status of multiple replicas in the distributed system and to identify replicas in the online incremental reconstruction state using reconstruction tags.

[0041] As an optional implementation of this application, the deletion unit is further configured to obtain a second target replica in the distributed system; the second target replica is a replica in a normal state in the distributed system; determine the target object where the data block to be deleted is located in the second target replica according to the target index key, and perform a deletion operation on the target index key in the target object; after deleting the target index key, check the target object; if there are no other valid index keys in the target object, configure a normal deletion timestamp for the target object; the normal timestamp is the timestamp corresponding to the current time of the distributed system when it is found that there are no other valid data blocks in the target object; when the distributed system background traverses each object in the second target replica, obtain the latest operation timestamp corresponding to the target object; if the latest operation timestamp corresponding to the target object is the normal deletion timestamp, then delete the target object and reclaim space.

[0042] As an optional implementation of this application, the processing unit is further configured to, when performing a write operation on the target object, record the write operation timestamp corresponding to the time of the write operation, and configure the write operation timestamp for the target object.

[0043] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to cause the electronic device to implement the space reclamation method for online incremental reconstruction process described in any of the above embodiments when executing the computer program.

[0044] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computing device, causes the computing device to implement the space reclamation method for online incremental reconstruction as described in any of the above embodiments.

[0045] The space reclamation method for online incremental reconstruction provided in this application embodiment specifically includes: receiving a deletion instruction for a data block to be deleted issued by a client, and obtaining the target index key corresponding to the data block to be deleted according to the deletion instruction; obtaining a first target replica in the distributed system; the first target replica is a replica in the distributed system that is in the online incremental reconstruction state; determining the target object where the data block to be deleted is located in the first target replica according to the target index key, and performing a deletion operation on the target index key in the target object; the target object includes multiple [objects] including the target index key. Index key; after deleting the target index key, the target object is checked. If no other valid index key exists in the target object, a minimal deletion timestamp is configured for the target object; the minimal deletion timestamp is the first timestamp at the start of Greenwich Mean Time; when the distributed system backend traverses each object in the first target replica, it obtains the latest operation timestamp corresponding to the target object; if there exists a minimal deletion timestamp that is the latest operation timestamp of the target object, the target object is deleted and its space is reclaimed; if there exists a write operation timestamp greater than the minimal deletion timestamp, the target object is retained.

[0046] The beneficial effects of this application are as follows: To avoid data loss in online incremental reconstruction scenarios due to misidentification and deletion timestamps caused by partial key migration, this application configures a minimal deletion timestamp for the target object. Deletion and space reclamation operations are only performed when this minimal deletion timestamp matches the latest operation timestamp of the target object. This avoids the situation where objects are mistakenly deleted due to incorrect judgment that there are no valid keys under the object, greatly reducing the risk of data loss and improving the data reliability and stability of the distributed storage framework in complex scenarios. Furthermore, for the special scenario of online incremental reconstruction, this application accurately identifies the state of the target object, avoiding misjudgments during key migration, ensuring data integrity and availability in complex scenarios, and enabling the distributed storage framework to better adapt to complex and ever-changing business scenarios. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings that need to be called in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is one of the flowcharts of a space reclamation method in the online incremental reconstruction process provided in this application embodiment;

[0050] Figure 2 This is the second flowchart of a space reclamation method for online incremental reconstruction provided in this application embodiment;

[0051] Figure 3 A schematic diagram illustrating the framework of a space reclamation method for online incremental reconstruction provided in this application embodiment;

[0052] Figure 4 The third step of the space reclamation method in the online incremental reconstruction process provided in this application embodiment;

[0053] Figure 5 This is a schematic diagram of the structure of a space reclamation device for online incremental reconstruction process provided in an embodiment of this application;

[0054] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0055] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0056] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0057] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. Furthermore, in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0058] It should be noted that, in this document, the term "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0059] Metadata contains detailed index information about objects and data blocks, such as object name, data block key, offset, and length. In a distributed storage environment, data is scattered across numerous nodes. When specific data needs to be accessed, the system can quickly determine the data's location based on these indexes in the metadata, avoiding blindly searching the entire distributed storage space and greatly improving the speed and efficiency of data retrieval. For example, in a large-scale distributed file storage system, the file path and data block location information in the metadata can quickly locate the file content needed by the user, reducing data access latency.

[0060] This application provides a space reclamation method for online incremental reconstruction processes, referring to... Figure 1 As shown, the space reclamation method for the online incremental reconstruction process includes the following steps S101-S105:

[0061] S101. Receive the deletion instruction for the data block to be deleted sent by the client, and obtain the target index key corresponding to the data block to be deleted according to the deletion instruction.

[0062] In some embodiments, when the distributed system receives a deletion instruction for a data block to be deleted from the client, it immediately parses the instruction to obtain the target index key corresponding to the data block to be deleted, and obtains the storage location of the data block in the distributed system based on the index key, thereby accurately locating the data that needs to be deleted.

[0063] S102. Obtain the first target copy in the distributed system.

[0064] The first target replica is a replica in the distributed system that is in an online incremental reconstruction state.

[0065] In some embodiments of a distributed system, replication refers to redundant storage of data, where a single copy of data is stored on multiple different nodes. The primary purpose of this is to improve the system's reliability, fault tolerance, and performance. For example, in a distributed database system, a copy of the data may be stored on different servers. When one server fails, the copies on the other servers can still provide data services, preventing data loss and service interruption.

[0066] Since the embodiments of this application mainly address the issue of avoiding data loss when receiving a deletion instruction for a data block to be deleted, for replicas in the online incremental reconstruction state, this step requires first identifying the replicas in the distributed system that are in the online incremental reconstruction state, and then using the replicas in the online incremental reconstruction state as the first target replica.

[0067] In a distributed storage system, an online incremental reconstruction state refers to a replica that is gradually restoring its data integrity after a node (such as a hard drive or server) fails. In this state, the replica is not completely unavailable, but partially available, and is recovering lost data through incremental recovery—that is, only recovering data objects that changed during the failure. Compared to full recovery, this method utilizes resources more efficiently and reduces recovery time and impact on system performance.

[0068] S103. Determine the target object where the data block to be deleted is located in the first target copy according to the target index key, and perform a deletion operation on the target index key in the target object.

[0069] In distributed systems, metadata indexes typically use the format object name-object key-object offset-len (offset and length). This format clearly identifies the location and size of specific data blocks within a stored object, such as object1-key1-offset1-len1. Once the target index key is obtained, the target object containing the data block to be deleted can be determined in the first target replica based on the target index key, and the deletion operation can be performed on the target object using the target index key.

[0070] Specifically, the primary function of marking deletion timestamps in distributed systems is to achieve delayed deletion. When a client initiates a deletion operation, the system marks the key corresponding to the data block to be deleted with a deletion timestamp, instead of immediately performing physical deletion. This allows the client to quickly receive feedback that the deletion operation has been accepted, significantly reducing the latency of client deletion operations. After initiating a deletion operation, the user will soon see a system notification indicating successful deletion, without having to wait a long time for the data to be completely removed from the storage device. Furthermore, in distributed systems, data may be accessed and manipulated by multiple clients simultaneously. Marking deletion timestamps can serve as a means of ensuring data consistency. Before a data block is actually deleted, the system can determine the status of the data block based on the timestamp and coordinate other operations (such as reading and updating).

[0071] Subsequently, when traversing the target objects in the first target copy through the background aggregation process, those data blocks marked with deletion timestamps are deleted.

[0072] Specifically, the background aggregation process is a processing mechanism that runs in the background. It starts periodically or under specific conditions and is specifically responsible for processing data blocks marked with deletion timestamps. The background aggregation process finds the corresponding actual data block based on the key recorded in the metadata. For example, in a distributed storage system, the key can be used to locate the specific node and location where the data block is stored. Then, the background aggregation process performs the actual data block deletion operation and reclaims the freed-up storage space for later reuse.

[0073] S104. After deleting the target index key, check the target object. If there are no other valid index keys in the target object, configure a minimal deletion timestamp for the target object.

[0074] The minimal deletion timestamp is the first timestamp of the start of Greenwich Mean Time.

[0075] It's important to note that deleting the target index key is only the first step in the data deletion process of a distributed system. Further inspection of the target object and configuring extremely small deletion timestamps under specific conditions allow for more granular management of system resources and data objects. In distributed systems, data is organized in various forms across different objects, each containing multiple index keys and their corresponding data. After deleting a key, determining whether the target object still has other valid data helps decide whether to completely clean up the object, avoiding unnecessary resource consumption.

[0076] After deleting the target index key, the system checks the target object to determine if any other valid index keys still exist. A "valid index key" typically refers to a data identifier that is still associated with meaningful data and is needed during system operation. For example, in a distributed system storing user files, each file may consist of multiple data blocks, each identified by an index key.

[0077] In accordance with the embodiments of this application, when the first target copy is in an online incremental reconstruction state, after deleting the target index key in the target object in the first copy, there may be no other index keys in the target object, that is, there are no other valid index keys. In this case, since the first target copy is in an online incremental reconstruction state, it is also possible that the index keys in the target object have not been migrated, creating the illusion that there are no other index keys in the current target object, which is easy to be mistakenly deleted; it is also possible that there are indeed no other index keys in the current target object, that is, the current target object can be recycled.

[0078] Therefore, to avoid the situation where the index key in the target object is mistakenly deleted because it has not been migrated, this application configures a minimal deletion timestamp when the replica is in an online incremental reconstruction state. This minimal deletion timestamp is defined as the first timestamp at the start of Greenwich Mean Time (i.e., January 1, 1970, 00:00:00 UTC). This specific time point is chosen as the minimal deletion timestamp; thus, this minimal deletion timestamp, with its clear and minimal time identifier, facilitates judgment in subsequent processing, indicating that the object has entered the final "pending deletion" stage. Specifically, the specific checking method for the target object after deleting the target index key depends on the target object's data structure. If the target object is a simple hash table structure, it may be possible to traverse the key-value pairs of the hash table to check for the existence of keys other than the deleted key. If it is a more complex tree structure, it may be necessary to check the index keys in the nodes according to a tree traversal algorithm (such as depth-first traversal or breadth-first traversal).

[0079] S105. When the distributed system backend is traversing each object in the first target replica, it obtains the latest operation timestamp corresponding to the target object.

[0080] In the daily maintenance of distributed systems, the backend periodically or irregularly traverses each object in the first target replica. This process is a crucial step for the system to check and optimize its own storage state. By traversing the system, the current state of each object can be understood, and based on different timestamps, the appropriate operation can be determined for the target object, ensuring the effective utilization of system storage resources and data integrity.

[0081] Specifically, when traversing objects in the first target replica in the background of the distributed system, the latest operation timestamp for each target object must first be determined. A timestamp is a marker recording the time when data operations occur in the system; for each target object, it records the time of the most recent operation performed on that object. This method of obtaining operation timestamps helps the system understand the object's state change history. For example, in a distributed file storage system, each time a write or delete operation is performed on a file (target object), the corresponding timestamp is updated. By obtaining the latest operation timestamp and then determining the relationship between the latest operation timestamp and the smallest deletion timestamp, it can be determined whether the target object needs to be deleted and reclaimed subsequently.

[0082] S106. If there exists a minimum deletion timestamp that is the latest operation timestamp of the target object, then the target object is deleted and its space is reclaimed.

[0083] Specifically, the minimal deletion timestamp is a special timestamp configured when the target object has no other valid index keys, indicating that the object is essentially "useless" and awaiting final cleanup. When the system detects that the minimal deletion timestamp is the latest operation timestamp of the target object, it means that no new operations have changed the object's state since the timestamp was configured. Therefore, the target object actually has no valid index keys, meaning it does not contain valid data. At this point, the system can perform a deletion operation, removing the target object from storage and reclaiming the space it occupies for other data to use.

[0084] S107. If there is a write operation timestamp that is greater than the minimum deletion timestamp, then the target object is retained.

[0085] Specifically, the write operation timestamp records the time when a data write operation was performed on the target object. When the system detects that the write operation timestamp is greater than the minimum deletion timestamp, it indicates that new data was written after the object was marked for deletion, meaning that the index key in the target object has not been migrated completely. This is because a write operation was performed on the target object after it was marked with the minimum deletion timestamp, thus creating the write operation timestamp.

[0086] This means that the target object still contains valid data, and deleting it would result in data loss. Therefore, the target object needs to be retained.

[0087] Through this timestamp-based judgment mechanism, distributed systems can intelligently manage storage resources, ensuring data integrity while promptly cleaning up useless objects, thereby improving the system's storage efficiency and performance.

[0088] To avoid data loss in online incremental reconstruction scenarios due to misidentification and incorrect timestamp assignment of objects during key migration, this application configures a minimal deletion timestamp for the target object. Deletion and space reclamation operations are only performed when this minimal timestamp matches the latest operation timestamp of the target object. This avoids accidental deletion of objects due to incorrect judgment that there are no valid keys, significantly reducing the risk of data loss and improving the data reliability and stability of the distributed storage framework in complex scenarios. Furthermore, for the special scenario of online incremental reconstruction, this application accurately identifies the state of the target object, avoiding misjudgments during key migration and ensuring data integrity and availability in complex scenarios. This allows the distributed storage framework to better adapt to complex and ever-changing business scenarios.

[0089] As an extension and refinement of the above embodiments, refer to Figure 1As shown, another space reclamation method for online incremental reconstruction includes the following steps S201-S208:

[0090] S201. Receive the deletion instruction for the data block to be deleted sent by the client, and obtain the target index key corresponding to the data block to be deleted according to the deletion instruction.

[0091] S202. Obtain the status labels corresponding to multiple replicas in the distributed system, and use the replica with the status label being a reconstruction label as the first target replica.

[0092] In this embodiment, the reconstruction tag indicates that the replica is undergoing online incremental reconstruction. Online incremental reconstruction is a common maintenance operation in distributed systems, which may involve data reorganization, repairing damaged data blocks, and optimizing storage structures. For example, when the system detects that some data in a replica is partially corrupted, or when the data layout of the replica needs to be adjusted to improve data access performance, the replica will be reconstructed.

[0093] Since the embodiments of this application are for a space reclamation method in the process of online incremental reconstruction, it is necessary to first find the replica in the online incremental reconstruction state from multiple replicas in the distributed system, namely the first target replica.

[0094] Specifically, before obtaining the status tags corresponding to multiple replicas in the distributed system and using the replica with the status tag being a reconstruction tag as the first target replica, this embodiment of the application also needs to perform the following steps:

[0095] Identify the states of multiple replicas in the distributed system, and mark replicas in the online incremental reconstruction state with reconstruction tags.

[0096] It should be noted that in a distributed system, replicas are a form of redundant data storage, and different replicas may be in multiple states. To facilitate special handling of replicas in the online incremental reconstruction state, it is necessary to first identify the states of multiple replicas in the distributed system and mark replicas in the online incremental reconstruction state with reconstruction tags.

[0097] This helps the system to manage replicas differently, especially when performing operations such as data deletion, allowing for specific subsequent strategies to be adopted for replicas in that special state.

[0098] Reference Figure 3 As shown, Figure 3Replica 1.2 is marked with a special flag: rebuilding; this indicates that replica 1.2 is in an online incremental reconstruction state, while replicas 1.1 and 1.3 are in an offline incremental reconstruction state.

[0099] S203. Determine the target object where the data block to be deleted is located in the first target copy according to the target index key, and mark the target index key with a deletion tag.

[0100] The deletion tag is used to instruct the distributed system to delete the data block to be deleted corresponding to the target index key in the background.

[0101] In some embodiments of a distributed system, data is typically stored and organized using a data structure such as a hash table, tree structure, or object storage. When a target index key is obtained, the system needs to search for the target object corresponding to that key in the first target replica. Then, a deletion tag is set for the target index key within that target object. The main purpose of this deletion tag is to delay the deletion operation, as directly deleting data can impact system performance and consistency, especially in a distributed environment involving multiple replicas and complex operational processes. By setting deletion tags, the system can postpone the deletion operation to the background, allowing client operations to complete faster without waiting for the actual deletion to finish. This avoids prolonged lock waits or network latency, improving system response speed and user experience.

[0102] Specifically, the implementation of tagging the target index key with a deletion tag depends on the data storage structure. If the target object is a simple dictionary, a special key-value pair can be added to the dictionary entry storing the target index key to represent the deletion tag.

[0103] S204. When the distributed system backend traverses the target object in the first target replica, it deletes the data block to be deleted corresponding to the target index key in the target object according to the deletion tag on the target index key.

[0104] After step S203 is executed, when the distributed system traverses all keys in the target object in the first target replica in the background, it can delete the data block to be deleted corresponding to the target index key in the target object according to the deletion tag on the target index key.

[0105] It should be noted that, in the embodiments of this application, the traversal operation of the distributed system can be triggered by a timer, such as setting a timer so that the background program starts and checks the replicas at regular intervals; or it can be triggered by an event, such as when the system load is low or when certain specific conditions are met.

[0106] S205. After deleting the target index key, check the target object. If there are no other valid index keys in the target object, configure a minimal deletion timestamp for the target object.

[0107] The minimal deletion timestamp is the first timestamp of the start of Greenwich Mean Time.

[0108] S206. When the distributed system backend is traversing each object in the first target replica, it obtains the latest operation timestamp corresponding to the target object.

[0109] S207. If there exists a minimum deletion timestamp that is the latest operation timestamp of the target object, then the target object is deleted and its space is reclaimed.

[0110] S208. If there is a write operation timestamp greater than the minimum deletion timestamp, then the target object is retained.

[0111] In this embodiment of the application, if there is a write operation timestamp that is greater than the minimum deletion timestamp, it indicates that during the current online incremental reconstruction process, in the target object in the first copy, there is a phenomenon where the index key has not been migrated but has been marked with a minimum deletion timestamp, and after the target object is marked with a minimum deletion timestamp, a write operation is performed in the target object.

[0112] Therefore, the embodiments of this application also require the following steps:

[0113] When performing a write operation on the target object, the write operation timestamp corresponding to the time of the write operation is recorded, and the write operation timestamp is configured for the target object.

[0114] By following the steps above, the write operation timestamp corresponding to the write operation is recorded, which facilitates subsequent comparison with the minimum deletion timestamp and avoids the situation of accidentally deleting the target object.

[0115] As an extension and refinement of the above embodiments, this application embodiment describes that after receiving the deletion instruction of the data block to be deleted issued by the client, and obtaining the target index key corresponding to the data block to be deleted according to the deletion instruction, when the replica status is normal, the following steps S401-S405 need to be performed:

[0116] S401. Obtain the second target copy in the distributed system.

[0117] The second target copy is a copy in a normal state in the distributed system.

[0118] Unlike the embodiments provided in steps S101-S107 above, the embodiments of this application mainly illustrate how to handle the deletion instructions for data blocks to be deleted issued by the client for replicas in the distributed system that are in a normal state.

[0119] Specifically, a replica in a normal state indicates that it is not in an online incremental reconstruction state, can normally receive and process various operations, including data writing and deletion operations, and has no ongoing abnormal operations or maintenance tasks.

[0120] S402. Determine the target object where the data block to be deleted is located in the second target copy according to the target index key, and perform a deletion operation on the target index key in the target object.

[0121] In this embodiment of the application, similar to the previous operation in the first target copy, it is also necessary to find the target object where the data block to be deleted is located in the second target copy based on the target index key.

[0122] S403. After deleting the target index key, check the target object. If there are no other valid index keys in the target object, configure a normal deletion timestamp for the target object.

[0123] The normal timestamp is the timestamp corresponding to the current time of the distributed system when it is found that there are no other valid data blocks in the target object.

[0124] Specifically, in the second target copy, when there are no other valid index keys in the target object, a normal deletion timestamp is configured for the target object. The normal deletion timestamp is the time when it is found that there are no other valid data blocks in the target object, and it can be generated using the current system time.

[0125] S404. When the distributed system backend is traversing each object in the second target replica, it obtains the latest operation timestamp corresponding to the target object.

[0126] Then, when traversing each object in the second target replica in the background of the distributed system, the system further determines whether to delete the target object and reclaim its space based on the relationship between the latest operation timestamp of the target object and the normal deletion timestamp.

[0127] S405. If the latest operation timestamp corresponding to the target object is the normal deletion timestamp, then the target object is deleted and its space is reclaimed.

[0128] Specifically, when the latest operation timestamp of the target object is the normal deletion timestamp, it indicates that there are no other valid index keys in the target object, that is, there is no valid data, and the target object is then deleted and its space is reclaimed.

[0129] Furthermore, the biggest difference between the first target replica in the online incremental reconstruction state and the second target replica in the normal state after deleting the target index key is that the configured deletion timestamps are different when there are no other valid keys in the target object. The first target replica is configured with a minimal deletion timestamp (the first timestamp starting from Greenwich Mean Time), while the second target replica is configured with a normal deletion timestamp, which is the timestamp corresponding to when it is found that there are no other valid data blocks in the target object.

[0130] This is because the first target replica is in an online incremental reconstruction state, and the system cannot immediately determine whether the data can be safely deleted. By setting a very small deletion timestamp, the system will determine whether to delete the object based on this special timestamp and other conditions while traversing the replicas in the background. This postpones the data deletion operation, giving the system enough time to ensure the integrity and consistency of the data during the reconstruction process. The second target replica, on the other hand, is in a normal state and does not require the complex reconstruction process of the first target replica. Therefore, it will not encounter the phenomenon of the index key not being migrated. Thus, when traversing objects in the replicas in the distributed system, it can directly and relatively quickly determine whether a data block can be deleted based on the normal deletion timestamp.

[0131] Based on the same inventive concept, as an implementation of the above method, this application embodiment also provides a space recycling device for the online incremental reconstruction process. This embodiment corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the space recycling device for the online incremental reconstruction process in this embodiment can correspondingly implement all the contents of the aforementioned method embodiment.

[0132] This application provides a space reclamation device for online incremental reconstruction processes. Figure 5 This is a schematic diagram of the space reclamation device for the online incremental reconstruction process, as shown below. Figure 5 As shown, the space reclamation device 500 for online incremental reconfiguration processes includes:

[0133] The receiving unit 501 is used to receive the deletion instruction of the data block to be deleted sent by the client, and to obtain the target index key corresponding to the data block to be deleted according to the deletion instruction of the data block to be deleted;

[0134] The acquisition unit 502 is used to acquire a first target replica in the distributed system; the first target replica is a replica in the distributed system that is in an online incremental reconstruction state;

[0135] The deletion unit 503 is used to determine the target object where the data block to be deleted is located in the first target copy according to the target index key, and to perform a deletion operation on the target index key in the target object;

[0136] The inspection unit 504 is used to inspect the target object after the target index key is deleted. If there is no other valid index key in the target object, a minimal deletion timestamp is configured for the target object. The minimal deletion timestamp is the first timestamp of the start of Greenwich Mean Time.

[0137] The traversal unit 505 is used to obtain the latest operation timestamp corresponding to the target object when the distributed system backend traverses each object in the first target replica.

[0138] The processing unit 506 is configured to delete the target object and reclaim space if the smallest deletion timestamp is the latest operation timestamp of the target object.

[0139] The processing unit 506 is further configured to retain the target object if there is a write operation timestamp greater than the minimum deletion timestamp.

[0140] As an optional implementation of this application, the deletion unit 503 is specifically used to determine the target object where the data block to be deleted is located in the first target replica according to the target index key, and to mark the target index key with a deletion tag; the deletion tag is used to instruct the distributed system to delete the data block to be deleted corresponding to the target index key in the background; when the distributed system in the background traverses the target object in the first target replica, it deletes the data block to be deleted corresponding to the target index key in the target object according to the deletion tag on the target index key.

[0141] As an optional implementation of this application, the acquisition unit 502 is specifically used to acquire the status tags corresponding to multiple replicas in the distributed system, and to use the replica with the status tag being a reconstruction tag as the first target replica.

[0142] As an optional implementation of this application, the acquisition unit 502 is further configured to identify the status of multiple replicas in the distributed system and to identify replicas in the online incremental reconstruction state using reconstruction tags.

[0143] As an optional implementation of this application, the deletion unit 503 is further configured to obtain a second target replica in the distributed system; the second target replica is a replica in a normal state in the distributed system; determine the target object where the data block to be deleted is located in the second target replica according to the target index key, and perform a deletion operation on the target index key in the target object; after deleting the target index key, check the target object; if there are no other valid index keys in the target object, configure a normal deletion timestamp for the target object; the normal timestamp is the timestamp corresponding to the current time of the distributed system when it is found that there are no other valid data blocks in the target object; when the distributed system background traverses each object in the second target replica, obtain the latest operation timestamp corresponding to the target object; if the latest operation timestamp corresponding to the target object is the normal deletion timestamp, then delete the target object and reclaim space.

[0144] As an optional implementation of this application, the processing unit 506 is further configured to record the write operation timestamp corresponding to the time of the write operation when performing a write operation on the target object, and configure the write operation timestamp for the target object.

[0145] Based on the same inventive concept, this disclosure also provides an electronic device. Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this disclosure, such as... Figure 6 As shown, the electronic device provided in this embodiment includes a memory 601 and a processor 602. The memory 601 is used to store a computer program; the processor 602 is used to execute the audio data processing method provided in the above embodiment when executing the computer program.

[0146] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it enables the computing device to implement the space reclamation method for online incremental reconstruction provided in the above embodiments.

[0147] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0148] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0149] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0150] Computer-readable media include both permanent and non-permanent, removable and non-removable storage media. Storage media can store information using any method or technology; the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for space reclamation during online incremental reconstruction, characterized in that, Applied to distributed systems, including: Receive the deletion instruction for the data block to be deleted sent by the client, and obtain the target index key corresponding to the data block to be deleted according to the deletion instruction; Obtain a first target replica in the distributed system; the first target replica is a replica in the distributed system that is in an online incremental reconstruction state; Based on the target index key, the target object containing the data block to be deleted is determined in the first target replica, and a deletion operation is performed on the target index key in the target object; the target object includes multiple index keys, including the target index key; After deleting the target index key, the target object is checked. If there are no other valid index keys in the target object, a minimal deletion timestamp is configured for the target object. The minimal deletion timestamp is the first timestamp of the start of Greenwich Mean Time. When the distributed system backend traverses each object in the first target replica, it obtains the latest operation timestamp corresponding to the target object; If there exists a minimum deletion timestamp that is the latest operation timestamp of the target object, then the target object will be deleted and its space reclaimed. If a write operation timestamp is greater than the minimum deletion timestamp, then the target object is retained.

2. The method according to claim 1, characterized in that, The step of determining the target object containing the data block to be deleted in the first target replica based on the target index key, and performing a deletion operation on the target index key in the target object, includes: Based on the target index key, the target object containing the data block to be deleted is determined in the first target replica, and a deletion tag is marked on the target index key; the deletion tag is used to instruct the distributed system to delete the data block to be deleted corresponding to the target index key in the background; When the distributed system backend traverses the target object in the first target replica, it deletes the data block to be deleted corresponding to the target index key in the target object according to the deletion tag on the target index key.

3. The method according to claim 1, characterized in that, Obtaining the first target replica in the distributed system includes: Obtain the status labels corresponding to multiple replicas in the distributed system, and use the replica with the status label being a reconstruction label as the first target replica.

4. The method according to claim 3, characterized in that, Before obtaining the status labels corresponding to multiple replicas in the distributed system and using the replica with the status label being a reconstruction label as the first target replica, the method further includes: Identify the states of multiple replicas in the distributed system, and mark replicas in the online incremental reconstruction state with reconstruction tags.

5. The method according to claim 1, characterized in that, After receiving the deletion instruction for the data block to be deleted from the client and obtaining the target index key corresponding to the data block to be deleted based on the deletion instruction, the method further includes: Obtain a second target replica in the distributed system; the second target replica is a replica in the distributed system that is in a normal state. Based on the target index key, determine the target object where the data block to be deleted is located in the second target replica, and perform a deletion operation on the target index key in the target object; After deleting the target index key, the target object is checked. If there are no other valid index keys in the target object, a normal deletion timestamp is configured for the target object. The normal deletion timestamp is the timestamp corresponding to the current time of the distributed system when it is found that there are no other valid data blocks in the target object. When the distributed system backend traverses each object in the second target replica, it obtains the latest operation timestamp corresponding to the target object; If the latest operation timestamp corresponding to the target object is the normal deletion timestamp, then the target object will be deleted and its space will be reclaimed.

6. The method according to claim 1, characterized in that, The method further includes: When performing a write operation on the target object, the write operation timestamp corresponding to the time of the write operation is recorded, and the write operation timestamp is configured for the target object.

7. A space reclamation device for online incremental reconfiguration processes, characterized in that, include: The receiving unit is used to receive the deletion instruction of the data block to be deleted sent by the client, and to obtain the target index key corresponding to the data block to be deleted according to the deletion instruction of the data block to be deleted; The acquisition unit is used to acquire the first target copy in the distributed system. The first target replica is a replica in the distributed system that is in an online incremental reconstruction state; The deletion unit is configured to determine the target object where the data block to be deleted is located in the first target copy based on the target index key, and perform a deletion operation on the target index key in the target object; The inspection unit is used to inspect the target object after the target index key is deleted. If there are no other valid index keys in the target object, a minimal deletion timestamp is configured for the target object. The minimal deletion timestamp is the first timestamp of the start of Greenwich Mean Time. The traversal unit is used to obtain the latest operation timestamp corresponding to the target object when the distributed system backend traverses each object in the first target replica; The processing unit is configured to delete the target object and reclaim space if the smallest deletion timestamp is the latest operation timestamp of the target object. The processing unit is further configured to retain the target object if there is a write operation timestamp greater than the minimum deletion timestamp.

8. The apparatus according to claim 7, characterized in that, The deletion unit is specifically used for: Based on the target index key, the target object containing the data block to be deleted is determined in the first target replica, and a deletion tag is marked on the target index key; the deletion tag is used to instruct the distributed system to delete the data block to be deleted corresponding to the target index key in the background; When the distributed system backend traverses the target object in the first target replica, it deletes the data block to be deleted corresponding to the target index key in the target object according to the deletion tag on the target index key.

9. An electronic device, characterized in that, include: A memory and a processor, the memory being used to store a computer program; the processor being used, when executing the computer program, to cause the electronic device to implement the space reclamation method for online incremental reconfiguration processes as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a computing device, causes the computing device to implement the space reclamation method for online incremental reconstruction as described in any one of claims 1-6.

Images