A method and apparatus for hard drive waste recycling

Abstract

This application relates to a method and apparatus for hard disk garbage collection. The method includes: constructing an index container hash tree, using the triplet of object identifier, data key, and attribute key as the hash key, storing record interval information and corresponding storage object identifiers, and caching high-frequency data using an LRU strategy; generating hash keys based on the metadata of the storage object to be reclaimed and searching the hash tree; based on the relationship between the record interval information and the current storage object identifier, executing three types of judgment rules: if the record intervals are consistent and the corresponding storage object identifier is small, it is directly judged as garbage; if it is large, a reverse lookup is performed and the cache is updated; if the intervals overlap or cover, judgment is made based on the size of the storage object identifier or a reverse lookup is performed; if the interval does not exist, a reverse lookup is performed directly; finally, valid data is migrated and garbage space is released based on the reverse lookup results. This method reduces reverse lookup RPC requests through hash caching, overcomes concurrency limitations, accelerates garbage data judgment, significantly improves GC collection efficiency, and ensures the stability of storage cluster capacity.

Description

Technical Field

[0001] This application relates to the field of hard disk waste disposal technology, and in particular to a hard disk waste recycling method, apparatus, computer-readable storage medium, and electronic device. Background Technology

[0002] In storage systems, to fully utilize hard drive write performance, the industry commonly employs Redirect-on-Write (ROW) technology. When a write request to a disk pool object is received, the system does not directly overwrite existing data. Instead, it redirects the new data and aggregates it into large-granularity input / output (IO) operations, ultimately writing them sequentially to the storage objects allocated by the disk pool. This technology is widely used in distributed storage because its sequential write characteristics significantly improve data write efficiency. However, the append-only nature of ROW technology can cause historical data in the same storage location to become invalid (i.e., "garbage data"), requiring garbage collection (GC) to reclaim this invalid space for resource reuse.

[0003] For large objects in ROW (Remote Overlay) technology (typically 128MB in size), existing GC (Garbage Collection) processes typically scan large objects: when the amount of garbage data within an object reaches a preset threshold, a GC migration operation is triggered—identifying and migrating valid data within the object to a newly allocated storage object, and releasing the invalid space of the original object after migration. Metadata reverse lookup is a core step in this process: by reverse-checking metadata information, it determines whether the data within the object is valid (if valid, it needs to be migrated; if invalid, it is released directly). However, concurrency control for metadata reverse lookup faces a critical challenge: if the concurrency is too high, it will lead to a large number of concurrent RPC (Remote Procedure Call) requests, causing network or server resource contention and RPC blocking; if the concurrency is too low, system resources cannot be fully utilized, resulting in low GC efficiency. Therefore, the GC efficiency of existing ROW technology is largely limited by the concurrency capability of metadata reverse lookup. In conclusion, developing a more efficient, reliable, and flexible disk garbage collection method has become an urgent need in this industry. Summary of the Invention

[0004] To address the aforementioned technical bottlenecks, this application proposes a novel hard disk garbage collection method. More specifically, this application proposes a GC collection method based on a hash caching mechanism, aiming to improve the overall performance of garbage collection by optimizing the concurrent processing efficiency of metadata reverse lookup.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] A first aspect of this application provides a hard disk garbage collection method, the method comprising the following steps:

[0007] S1: Construct an index container hash tree, wherein the hash tree uses a triplet of object identifier, data key, and attribute key as the hash key, and the record interval information and the corresponding storage object identifier as the hash value, wherein the record interval information includes the record interval range and the record type;

[0008] S2: Generate a hash key based on the metadata information of the storage object to be reclaimed, and use the hash tree to find the record range information corresponding to the metadata information;

[0009] S3: Based on the record interval information obtained in the previous step and the storage object identifier of the current storage object to be reclaimed, the validity of the data is determined according to the preset judgment rules;

[0010] S4: Based on the judgment result of the previous step, migrate the valid data to the new storage object and release the garbage space in the original storage object.

[0011] Optionally, in the method of this application, the hash value in step S1 includes the lower limit, upper limit, record type, and corresponding storage object identifier of the record interval; wherein, if the data is an array type, the lower limit and upper limit of the record interval are the actual boundaries of the array interval; if the data is a single-value type, the lower limit and upper limit of the record interval are both single-value values.

[0012] Optionally, in the method of this application, the following judgment rule is executed in step S3 to determine the validity of the data:

[0013] S31: If the record interval information exists and is consistent with the record interval of the metadata information:

[0014] When the storage object identifier corresponding to the record interval information is greater than the storage object identifier of the current storage object to be reclaimed, the data corresponding to the record interval information is determined to be garbage data;

[0015] When the storage object identifier corresponding to the record interval information is less than or equal to the storage object identifier of the current storage object to be reclaimed, a reverse lookup is initiated again and the storage object identifier in the hash tree is updated.

[0016] S32: If the record interval information exists but overlaps or covers the record interval of the metadata information:

[0017] When the storage object identifier corresponding to the recorded interval information is greater than the storage object identifier of the current storage object to be reclaimed, the data in the overlapping or overwritten portion is determined to be garbage data;

[0018] When the storage object identifier corresponding to the record interval information is less than or equal to the storage object identifier of the current storage object to be reclaimed, a reverse lookup is initiated to determine the validity of the data.

[0019] S33: If the record interval information does not exist, a reverse lookup is initiated directly to determine the validity of the data.

[0020] Optionally, in the method of this application, the step S31 of re-initiating the reverse lookup and updating the storage object identifier in the hash tree includes: obtaining the latest storage object identifier through the reverse lookup; if the latest storage object identifier is greater than the storage object identifier of the current storage object to be reclaimed, then updating the storage object identifier of the corresponding record in the hash tree to the latest storage object identifier.

[0021] Optionally, the method of this application also includes:

[0022] Step S5: Sort the record interval information in the reverse lookup results from smallest to largest according to the lower limit of the interval, so as to optimize the efficiency of subsequent interval processing.

[0023] Optionally, the method of this application also includes:

[0024] Step S6: When there are overlapping or overlapping record intervals in the reverse lookup results, split the record intervals according to the principle that the interval with the larger storage object identifier covers the interval with the smaller storage object identifier.

[0025] Optionally, the method of this application also includes:

[0026] Step S7: When there are consecutive record intervals with the same storage object identifier in the reverse lookup results, merge them into a larger record interval.

[0027] Optionally, in the method of this application, the hash tree in step S1 is a cached hash tree based on the Least Recently Used (LRU) strategy, which is used to prioritize the retention of record range information with high frequency access and reduce the number of reverse lookup requests.

[0028] A second aspect of this application provides a hard disk waste recycling device, the device comprising:

[0029] The hash tree construction module is used to construct an index container hash tree. The hash tree uses a triplet of object identifier, data key, and attribute key as the hash key, and record interval information and corresponding storage object identifier as the hash value. The record interval information includes the record interval range and record type.

[0030] The hash lookup module is used to generate a hash key based on the metadata information of the storage object to be reclaimed, and to look up the record range information corresponding to the metadata information through the hash tree.

[0031] The data validity determination module is used to determine the data validity based on the record range information obtained by the hash lookup module and the storage object identifier of the current storage object to be reclaimed, and according to the preset determination rules.

[0032] The data processing module is used to migrate valid data to a new storage object based on the judgment result of the data validity judgment module, and release the garbage space in the original storage object.

[0033] The device performs the steps of the aforementioned hard disk garbage collection method during operation.

[0034] Furthermore, the device of this application also includes:

[0035] The record interval sorting module is used to sort the record interval information in the reverse lookup results from smallest to largest according to the lower limit of the interval, so as to optimize the efficiency of subsequent interval processing.

[0036] The record interval splitting module is used to split record intervals according to the principle that the interval with the larger storage object identifier overwrites the interval with the smaller storage object identifier when there are overlapping or overlapping record intervals in the reverse lookup results.

[0037] The record interval merging module is used to merge consecutive record intervals with the same storage object identifier into a larger record interval when there are such intervals in the reverse lookup results.

[0038] A third aspect of this application provides an electronic device, including: a memory and a processor;

[0039] Memory: Used to store computer programs;

[0040] Processor: Used to execute the computer program to implement the steps of the aforementioned hard disk garbage collection method.

[0041] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned hard disk garbage collection method.

[0042] In summary, the hash-caching-based disk garbage collection method proposed in this application achieves performance optimization and stability improvement at multiple technical levels by introducing a reverse hash cache mechanism. Specific advantages are as follows:

[0043] (1) Significantly reduce the overhead of reverse lookup RPC requests: The reverse lookup hash cache avoids repeatedly initiating a large number of RPC reverse lookup requests by pre-storing or dynamically caching key information of metadata, effectively reducing the load on network communication and server-side processing, and reducing the communication link latency of the reverse lookup process.

[0044] (2) Breaking through the limitations of reverse lookup concurrency: In traditional solutions, the concurrency of reverse lookup is limited by the processing capacity of RPC requests and resource competition. This solution localizes the metadata information of high-frequency reverse lookup through hash caching. A single reverse lookup operation can batch process the validity determination of more data objects. Large-scale data verification can be completed without relying on high-concurrency RPC requests, which significantly improves the efficiency of concurrent processing.

[0045] (3) Accelerate the efficiency of garbage data judgment: The fast query characteristics of hash cache greatly shorten the response time of data validity judgment, and can quickly screen the data in the target object in a short time, clearly distinguish between valid data and garbage data, and provide efficient data basis for subsequent GC relocation operations.

[0046] (4) Improve the overall efficiency of GC recycling and ensure the stability of cluster capacity: Through the coordinated optimization of the above links, the time ratio of the reverse lookup stage in the GC recycling process is significantly reduced, effectively shortening the single GC cycle and improving the release rate of garbage space; at the same time, the fast and efficient garbage recycling capability avoids the problem of cluster capacity shortage caused by space recycling lag, ensuring the long-term stable availability of storage cluster capacity.

[0047] In summary, this solution optimizes the entire reverse lookup process through a hash caching mechanism, achieving multi-dimensional improvements in communication overhead, concurrency capabilities, and decision efficiency, providing a more efficient and stable garbage collection solution for storage systems based on ROW technology.

[0048] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of this application, the accompanying drawings involved in the description of the embodiments will be briefly introduced below. It should be noted that the drawings only show some embodiments of this application. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0050] Figure 1 This is a topology diagram of the cluster in an embodiment of this application.

[0051] Figure 2 This is a flowchart illustrating the overall implementation process of the hard disk garbage collection method described in this application.

[0052] Figure 3This is a flowchart of the GC process for the stored objects in this application (with the addition of LRU hash cache checks).

[0053] Figure 4 This is a diagram showing the distribution structure of the index metadata in this application.

[0054] Figure 5 This is a schematic diagram of the index reverse lookup process in the embodiments of this application (it is necessary to send RPC to each shard of the metadata for querying. If the results retrieved from the three shards are consistent, it can be confirmed that it is garbage or data).

[0055] Figure 6 This diagram illustrates the rules for splitting and merging recx intervals in the hash cache in this embodiment. The horizontal axis represents recx, and the vertical axis represents the storage object ID. If the recx to be retrieved is between cache 1 and cache 2, following the principle of larger IDs covering smaller ones, the valid recx is split into three parts marked in red. The recx to be retrieved only needs to retrieve the part marked in red. After the retrieval is completed, the hash cache is updated, thus forming a larger recx cache (formed by splicing all the red lines). If a subsequent retrieval request falls within this interval, it can be determined whether it is garbage.

[0056] Figure 7 This is a structural diagram of the hard disk waste recycling device of this application.

[0057] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0059] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0060] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0061] Figure 2 The diagram shows the overall implementation process of the hard disk garbage collection method provided in this application, including the following steps:

[0062] S1: Construct an index container hash tree, wherein the hash tree uses a triplet of object identifier, data key, and attribute key as the hash key, and the record interval information and the corresponding storage object identifier as the hash value, wherein the record interval information includes the record interval range and the record type;

[0063] S2: Generate a hash key based on the metadata information of the storage object to be reclaimed, and use the hash tree to find the record range information corresponding to the metadata information;

[0064] S3: Based on the record interval information obtained in the previous step and the storage object identifier of the current storage object to be reclaimed, the validity of the data is determined according to the preset judgment rules;

[0065] S4: Based on the judgment result of the previous step, migrate the valid data to the new storage object and release the garbage space in the original storage object.

[0066] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0067] Assuming a 3-node cluster, with each node having 4 disks, the cluster topology is as follows: Figure 1As shown in the diagram, the index pool is mainly used to store metadata information corresponding to user data, while the data pool is used to store the actual user data. User data includes single-value (SV) and array-value types, where array values ​​are generally described in a fixed size, such as 512 bytes, e.g., [0,3], a total of 4 groups * 512 = When writing (2KB), it is first mapped to an index object, and then mapped to the corresponding master node (master refers to redundancy; for example, in a 3-replica architecture, one replica is chosen as the master node, and the other two are slave nodes). For example, the master node for index_objA1 / B1 / C1 is node0, the master node for index_objA2 / B2 / C2 is node1, and the master node for index_objA3 / B3 / C13 is node2. Then, on the master node, after aggregation writing, each storage object is requested. Assuming 3-replica redundancy, the requested storage objects are obj0 (tgt0, tgt4, tgt8), obj1 (tgt1, tgt5, tgt9), and obj2 (tgt2, tgt5, tgt9). That is, the write requests for A1 / B1 / C1 are aggregated together through obj0, the write requests for A2 / B2 / C2 are aggregated together through obj1, and the write requests for A3 / B3 / C3 are aggregated together through obj3. After aggregation, the metadata information of the index object (such as the index container `cont`, the index `obj` object ` / dkey / akey / recx`, and the backend storage address) is recorded and stored in the metadata space of the index pool; simultaneously, it is also recorded in `obj0` to locate its storage location within `obj0`. When this object repeatedly overwrites new `obj` storage objects, the data in the old `obj` storage objects becomes garbage data. When the amount of garbage in the storage objects reaches the GC reclamation threshold (e.g., a static threshold of 80%), the garbage space can be reclaimed using the GC garbage collection algorithm. Specifically, this involves reading the index metadata information recorded in the `obj` storage object and using this information for reverse lookup (in a distributed scenario, assuming the index object is a 3-replica object, it needs to be searched on the 3 replica shards of the object, sending 3 RPC read requests, involving 3 network transmissions, such as...). Figure 5 As shown in the figure, the reverse lookup result, that is, the metadata information of the index object (such as the index container cont, the index obj object / dkey / akey / recx, and the storage object ID), is compared with the information stored on the storage object to determine whether the data is garbage or real and valid data. If the reverse lookup result is valid data, the valid data needs to be moved to a new storage object.

[0068] In summary, due to the communication time and concurrency limitations of reverse lookup request RPC, the bottleneck of GC reclamation is limited by the reverse lookup step. Considering the write characteristics of ROW (later-allocated storage object IDs are larger), this application proposes a method to improve GC reclamation efficiency based on reverse lookup caching. Figure 3 The diagram shows the garbage collection (GC) process for stored objects in this application (with the addition of LRU hash cache checks). The specific implementation steps of this method are as follows:

[0069] 1. Construct the hash tree for the index container: Create a corresponding LRU hash table based on the index container, using the triplet obj, dkey, and akey as the hash key. The value includes recx[low, high] and the stored object ID. If the metadata represents an array type, then low and high are the actual range boundaries of recx; if it is SV data, then low and high are the values ​​of SV. The completed hash tree is as follows: Figure 4 As shown.

[0070] 2. Hash tree lookup: Generate hash keys (obj, dkey, akey) based on the index metadata information (index container, obj, dkey, akey, recx) and look up the recx range in the hash tree of the index container.

[0071] 3. Range Consistency Determination Rule: If the recx returned by the stored object exists in the hash table and the range is consistent, the determination rule is as follows:

[0072] 3.1 If the ID of the storage object where the reverse lookup recx is located is less than the ID of the storage object in the hash cache, it can be directly determined as garbage. For example, if obj with storage object ID 100 is being GCd, and a recx in its metadata is recorded as storage object ID 150 in the hash tree after reverse lookup, then the recx in storage object 100 must be garbage data.

[0073] 3.2 If the ID of the stored object is greater than or equal to the ID of the stored object in the cache, a reverse lookup is initiated again, and the hash cache is updated. For example, if an object with ID 100 is undergoing GC, and a piece of recx in its metadata is recorded as object ID 90 in the hash tree after a reverse lookup, it is impossible to determine whether recx is garbage or data in object 100. In this case, a reverse lookup needs to be initiated to confirm. If the stored object where the reverse lookup result is located is 120, then 120 is updated in the hash cache.

[0074] 4. Interval Overlap Determination Rules: If the recx retrieved from the stored object is inconsistent with or overlaps with the recx in the cache, the following rules shall be applied:

[0075] 4.1 For overlapping or overlapping parts, if the ID of the stored object is less than the ID of the cached stored object, it is judged as garbage. For example, if the stored object ID is 100 and obj is being GC, a segment recx[2,5] in its metadata can be found in the corresponding cache {recx[1,8],ObjId=101} through the hash table. Because the stored object ObjId in the cache is larger, recx[2,5] on the 100 object can be judged as garbage.

[0076] 4.2 For overlapping or overriding parts, when the ID of the storage object is greater than the ID of the cache storage object, a reverse lookup is required to confirm whether it is garbage or data. For example, if the storage object ID is 100 and obj is undergoing GC, a segment recx[2,5] in its metadata can be found in the corresponding cache {recx[1,8],ObjId=90} through the hash table. Because the storage object ObjId = 100 is relatively large, a reverse lookup is required to confirm whether it is garbage or data.

[0077] 5. Rule for determining if a range does not exist: If the recx lookup result for the stored object is not in the cache, then initiate a reverse lookup confirmation directly.

[0078] 6. Recx range sorting rules: The reverse lookup results are sorted by recx from smallest to largest. The range covered by the range is split. For example, if the stored values ​​of two recx ranges under the same akey in the hash tree are {recx[2,5],ObjId=100} and {recx[6,9],ObjId=90}, then the recx in the hash tree are sorted by the lower limit as {recx[2,5],ObjId=100} and {recx[6,9],ObjId=90}.

[0079] 7. recx interval splitting rules: such as Figure 6 As shown, following the principle of larger object IDs overwriting smaller ones, if the stored values ​​of two recx ranges under the same akey in the hash tree are {recx[2,5],ObjId=100} and {recx[3,9],ObjId=90}, then the recx in the hash tree will be split into {recx[2,5],ObjId=100} and {recx[6,9],ObjId=90} according to the principle of larger object IDs overwriting smaller ones.

[0080] 8. Recx range merging rule: When recx ranges are consecutive, if the stored object IDs are the same, they can be merged into a larger recx range. For example, if two recx ranges under the same akey have the stored values ​​{recx[2,5],ObjId=100} and {recx[6,9],ObjId=100} on the hash tree, they can be merged into one recx, namely {recx[2,9],ObjId=100}.

[0081] Figure 7 The image shows a hard disk waste recycling device according to this application, the device comprising:

[0082] The hash tree construction module is used to construct an index container hash tree. The hash tree uses a triplet of object identifier, data key, and attribute key as the hash key, and record interval information and corresponding storage object identifier as the hash value. The record interval information includes the record interval range and record type.

[0083] The hash lookup module is used to generate a hash key based on the metadata information of the storage object to be reclaimed, and to look up the record range information corresponding to the metadata information through the hash tree.

[0084] The data validity determination module is used to determine the data validity based on the record range information obtained by the hash lookup module and the storage object identifier of the current storage object to be reclaimed, and according to the preset determination rules.

[0085] The data processing module is used to migrate valid data to a new storage object based on the judgment result of the data validity judgment module, and release the garbage space in the original storage object.

[0086] The above-mentioned device implements the steps of the hard disk garbage collection method disclosed in this application when it is in operation.

[0087] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0088] like Figure 8As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the aforementioned hard disk garbage collection method.

[0089] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0090] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types:

[0091] Object-oriented programming languages, such as Java, Smalltalk, C++, etc.

[0092] Conventional procedural programming languages, such as "C" or similar programming languages.

[0093] The execution methods of program code include, but are not limited to:

[0094] It runs entirely on the user's computer;

[0095] Part of it executes on the user's computer, and part of it executes on a remote computer;

[0096] Execute as a standalone software package;

[0097] It is executed entirely on a remote computer or server.

[0098] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0099] Furthermore, this application also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the various steps of the hard disk garbage collection method disclosed in this application.

[0100] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following:

[0101] (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0102] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0103] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0104] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0105] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0106] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0107] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0108] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0109] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0110] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0111] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0112] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program contains program code for executing the hard disk garbage collection method disclosed in this application. When the computer program is executed by a processing device, it can achieve the functions defined in the embodiments of this application.

[0113] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0114] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A hard disk garbage collection method characterized by, The method comprises the following steps: S1: constructing an index container hash tree, the hash tree taking a triple of an object identifier, a data key, and an attribute key as a hash key, and taking record interval information and a corresponding storage object identifier as a hash value, wherein the record interval information comprises a record interval range and a record type; S2: generating a hash key according to metadata information of a storage object to be recycled, and finding record interval information corresponding to the metadata information through the hash tree; S3: determining data validity based on the record interval information found in the previous step and the storage object identifier of the current storage object to be recycled according to a preset determination rule; the following determination rule is executed in this step to determine data validity: S31: if the record interval information exists and is consistent with the record interval of the metadata information: when the storage object identifier corresponding to the record interval information is greater than the storage object identifier of the current storage object to be recycled, it is determined that the data corresponding to the record interval information is garbage data; when the storage object identifier corresponding to the record interval information is less than or equal to the storage object identifier of the current storage object to be recycled, reinitiating a reverse search and updating the storage object identifier in the hash tree; S32: if the record interval information exists but there is intersection or overlap with the record interval of the metadata information: when the storage object identifier corresponding to the record interval information is greater than the storage object identifier of the current storage object to be recycled, it is determined that the data in the intersection or overlap part is garbage data; when the storage object identifier corresponding to the record interval information is less than or equal to the storage object identifier of the current storage object to be recycled, reinitiating a reverse search to determine data validity; S33: if the record interval information does not exist, directly initiating a reverse search to determine data validity; S4: migrating valid data to a new storage object according to the determination result in the previous step, and releasing garbage space in the original storage object.

2. The method of claim 1, wherein, The hash value in step S1 comprises a lower limit, an upper limit, a record type, and a corresponding storage object identifier of the record interval; if the data is an array type, the lower limit and the upper limit of the record interval are the real boundaries of the array interval; if the data is a single value type, the lower limit and the upper limit of the record interval are both numerical values of the single value.

3. The method of claim 1, wherein, The reinitiation of the reverse search and the updating of the storage object identifier in the hash tree in step S31 comprises: obtaining the latest storage object identifier through the reverse search, and if the latest storage object identifier is greater than the storage object identifier of the current storage object to be recycled, updating the storage object identifier corresponding to the record in the hash tree to the latest storage object identifier.

4. The method of claim 1, wherein, The method further comprises: Step S5: sorting the record interval information in the reverse search result according to the interval lower limit from small to large, to optimize the subsequent interval processing efficiency.

5. The method of claim 1, wherein, The method further comprises: Step S6: when there are intersected or overlapped record intervals in the reverse search result, splitting the record intervals according to the principle that the storage object identifier of a larger interval covers the storage object identifier of a smaller interval.

6. The method of claim 1, wherein, The method further comprises: Step S7: when there are continuous and consistent record intervals in the reverse search result, merging them into a larger record interval.

7. The method of claim 1, wherein, The hash tree in step S1 is a cache hash tree based on a least recently used strategy, which is used to preferentially retain high-frequency accessed record interval information and reduce the number of reverse lookup requests.

8. A hard disk garbage collection apparatus characterized by comprising: The device comprises: a hash tree construction module, configured to construct an index container hash tree, the hash tree taking a triple of an object identifier, a data key and an attribute key as a hash key, and taking record interval information and a corresponding storage object identifier as a hash value, wherein the record interval information comprises a record interval range and a record type; a hash lookup module, configured to generate a hash key according to metadata information of a storage object to be recycled, and to find record interval information corresponding to the metadata information through the hash tree; a data validity determination module, configured to determine data validity based on the record interval information obtained by the hash lookup module and a storage object identifier of a current storage object to be recycled, and according to a preset determination rule; the module executes the following determination rule to determine data validity: (1) if the record interval information exists and is consistent with the record interval of the metadata information: when the storage object identifier corresponding to the record interval information is greater than the storage object identifier of the current storage object to be recycled, it is determined that the data corresponding to the record interval information is garbage data; when the storage object identifier corresponding to the record interval information is less than or equal to the storage object identifier of the current storage object to be recycled, reverse lookup is reinitiated and the storage object identifier in the hash tree is updated; (2) if the record interval information exists but intersects or covers the record interval of the metadata information: when the storage object identifier corresponding to the record interval information is greater than the storage object identifier of the current storage object to be recycled, it is determined that the data in the intersected or covered part is garbage data; when the storage object identifier corresponding to the record interval information is less than or equal to the storage object identifier of the current storage object to be recycled, reverse lookup is reinitiated to determine data validity; (3) if the record interval information does not exist, reverse lookup is directly initiated to determine data validity; a data processing module, configured to migrate valid data to a new storage object and release garbage space in the original storage object according to the determination result of the data validity determination module.

9. The apparatus of claim 8, wherein, The device further comprises: a record interval sorting module, configured to sort the record interval information in the reverse lookup result in ascending order of lower limit of interval, so as to optimize subsequent interval processing efficiency; a record interval splitting module, configured to split the record interval when there are intersected or covered record intervals in the reverse lookup result, according to the principle that a storage object identifier of a larger interval covers a storage object identifier of a smaller interval; a record interval merging module, configured to merge record intervals when there are continuous and consistent storage object identifiers in the reverse lookup result, into larger record intervals.

Images