A hard disk garbage collection method, device and electronic equipment

Abstract

The application discloses a hard disk garbage collection method and device, which is applied to a distributed storage system using a ROW technology. The method comprises the following steps: managing storage objects according to garbage quantity proportioning and preferentially collecting high-garbage-quantity objects; monitoring cluster capacity utilization, performing normal moving collection when the capacity utilization is low, and switching to moving-free collection based on garbage data continuous feature statistics when the capacity utilization is high; identifying continuous garbage data segments, and performing punching release when the length reaches an adaptively adjusted continuous garbage collection threshold; and correcting the garbage quantity of the punched objects and re-bucketing. Through adaptive matching of the GC strategy and the cluster capacity, the application avoids the risk of space exhaustion caused by moving writing in the high-capacity water level scene, effectively improves the garbage collection efficiency, and guarantees the continuity of the foreground business writing and the stability of the cluster capacity.

Description

Technical Field

[0001] This application relates to the field of storage system waste recycling technology, specifically to a hard disk waste recycling method, apparatus, and electronic equipment. Background Technology

[0002] In distributed storage systems, to fully utilize hard drive write performance, existing technologies commonly employ a redirect-on-write (ROW) mechanism. This mechanism redirects data to be written to objects in the storage pool to the same region and aggregates it, forming large-granularity input / output (I / O) before sequentially writing it to the objects allocated in the storage pool. Simultaneously, to further improve data write efficiency, the industry widely adopts append-only write. However, append-only write strategies result in a large amount of invalid garbage data from historical versions of the same data block after multiple updates, necessitating garbage collection (GC) to reclaim storage space. Current mainstream GC solutions are based on large object (e.g., 128MB) management granularity. By scanning objects and determining whether their garbage percentage reaches a preset threshold, they decide whether to perform a data migration operation (GCMigration)—that is, reading the valid data from the object to be reclaimed and writing it to a new object, then releasing the physical space occupied by the original object.

[0003] The aforementioned data migration process involves two stages: disk reading and writing. It encompasses multiple stages, including network transmission, disk I / O, and concurrent resource scheduling. Its efficiency is limited by network bandwidth, disk throughput, and competition for system concurrent resources, directly constraining the overall efficiency of garbage collection (GC). More critically, when the cluster's disk storage space approaches saturation, the incremental writes generated by GC data migration will continuously consume remaining storage capacity, creating a negative feedback loop between space reclamation and write consumption. This prevents timely release of garbage space, leading to write blockages in front-end services and severely impacting system availability and business continuity.

[0004] To address the aforementioned technical challenges, there is an urgent need to develop an optimization method that can dynamically adjust GC strategies based on cluster capacity status. This would improve GC efficiency and mitigate the risk of business blockage when storage space is at a critical point, while ensuring the stable operation of the storage system. Summary of the Invention

[0005] To address the shortcomings of existing GC strategies, such as lack of adaptability to cluster capacity status, low recycling efficiency in high-capacity scenarios, and potential exacerbation of space constraints, this application proposes a novel hard disk garbage collection method and apparatus. More specifically, this application proposes a GC strategy optimization method based on adaptive adjustment of cluster capacity. By dynamically sensing the remaining capacity of the storage pool and adjusting the GC triggering conditions, relocation granularity, or recycling intensity accordingly, this method aims to maximize garbage space recycling efficiency and improve overall system performance and reliability while ensuring the continuity of front-end business writes.

[0006] Specifically, this application provides the following technical solutions:

[0007] The first aspect of this application provides a hard disk garbage collection method, which is applied to a distributed storage system employing Redirect-on-Write (ROW) technology, such as... Figure 1 As shown, it includes the following steps:

[0008] S1. Waste volume bin management: Divide the waste volume of stored objects into multiple ranges and create corresponding recycling bins, and perform recycling in order of priority from high to low waste volume;

[0009] S2. Object binning and garbage quantity statistics: Calculate the proportion of garbage quantity of objects to be recycled and classify them into the corresponding recycling bins;

[0010] S3, GC strategy dynamic switching: Monitor cluster capacity utilization. When the capacity utilization is lower than the preset threshold, perform GC collection based on data migration; when the capacity utilization reaches or exceeds the preset threshold, switch to a non-migration GC collection method based on continuous characteristics of garbage data.

[0011] S4. Handling-free garbage collection: Based on the metadata and bitmap information of the stored object, identify continuous garbage data segments. When the total length of continuous garbage data reaches the continuous garbage collection threshold, perform a punching operation on the continuous garbage data segments to release space and mark the corresponding metadata as punched.

[0012] S5. Adaptive adjustment of continuous garbage collection threshold: The continuous garbage collection threshold is adaptively adjusted according to the urgency of cluster capacity utilization. The threshold is set according to the stripe alignment principle of the redundancy strategy.

[0013] S6. Waste Quantity Correction and Bin Switching: For storage objects with punched holes to release space, correct their waste quantity ratio and redistribute them to the corresponding recycling bins according to the corrected waste quantity ratio.

[0014] Furthermore, in the method of this application, in step S1, the proportion of the amount of waste is divided into tiers of 5%, and a total of 20 recycling bins are established, namely [0,5), [5,10), ..., [95,100].

[0015] Furthermore, in the method of this application, in step S2, the method for calculating the proportion of garbage data in the storage object to be recycled is as follows: using a bitmap marking algorithm, valid data and garbage data are marked at the smallest writing unit of the storage object, the number of garbage data marking bits in the bitmap is counted, and the proportion of garbage data is calculated.

[0016] Furthermore, in the method of this application, in step S3, the preset threshold is 85%;

[0017] The GC collection based on data migration includes: determining the valid data in the storage object through reverse lookup of metadata, reading the valid data from the old storage object and writing it to the new storage object, and deleting the old storage object after the data migration is completed.

[0018] Furthermore, in the method of this application, step S4, the method for identifying continuous garbage data segments includes: determining the position and length of each garbage data segment based on the metadata information of the stored object, determining whether adjacent garbage data segments are continuous, and if they are continuous, merging them to calculate the total length.

[0019] Furthermore, in the method of this application, in step S5, the redundancy strategy includes replica redundancy and erasure coding redundancy; for the erasure coding redundancy strategy, the continuous garbage collection threshold is the product of the baseline collection threshold and the number of stripe fragments; for the replica redundancy strategy, the continuous garbage collection threshold is equal to the baseline collection threshold.

[0020] Furthermore, in the method of this application, the baseline reclamation threshold is adaptively adjusted according to the urgency of cluster capacity utilization, specifically as follows:

[0021] When the capacity utilization rate is in the range of [85%, 90%), the baseline recovery threshold is 2M;

[0022] When the capacity utilization rate is in the range of [90%, 95%), the baseline recovery threshold is 1M;

[0023] When the capacity utilization rate is in the range of [95%, 100%], the baseline recovery threshold is 32K.

[0024] Furthermore, in the method of this application, in step S6, the method for correcting the garbage amount ratio is as follows: subtract the amount of space released by punching holes from the original garbage amount of the stored object to obtain the corrected garbage amount, and then calculate the corrected garbage amount ratio.

[0025] Furthermore, the method of this application also includes: during subsequent GC scanning, skipping the repeated counting and release of punched space based on the punched markings in the metadata.

[0026] A second aspect of this application provides a hard disk garbage collection device, which is applied to a distributed storage system employing redirected write technology and implements the steps of the aforementioned hard disk garbage collection method during operation, such as... Figure 2 As shown, the device includes:

[0027] The garbage bin module is used to divide the garbage amount of stored objects into multiple ranges and create corresponding recycling bins, and perform recycling in order of priority from high to low garbage amount.

[0028] The object binning module is used to calculate the percentage of garbage objects to be recycled and classify them into the corresponding recycling bins.

[0029] The strategy switching module is used to monitor the cluster capacity utilization. When the capacity utilization is lower than the preset threshold, it triggers GC reclamation based on data migration; when the capacity utilization reaches or exceeds the preset threshold, it switches to a non-migration GC reclamation method based on continuous feature statistics of garbage data.

[0030] The non-moving recycling module is used to identify continuous garbage data segments based on the metadata and bitmap information of the stored object. When the total length of the continuous garbage data reaches the continuous garbage recycling threshold, the module performs a punching operation on the continuous garbage data segments to release space and marks the corresponding metadata as punched.

[0031] The threshold adjustment module is used to adaptively adjust the continuous garbage collection threshold according to the urgency of cluster capacity utilization. The threshold is set according to the stripe alignment principle of the redundancy strategy.

[0032] The correction switching module is used to correct the garbage amount ratio of storage objects that have been punched to free up space, and then reallocate them to the corresponding recycling bins according to the corrected garbage amount ratio.

[0033] A third aspect of this application provides an electronic device, including: a memory and a processor;

[0034] Memory: Used to store computer programs;

[0035] Processor: Used to execute the computer program to implement the steps of the aforementioned hard disk garbage collection method.

[0036] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned hard disk garbage collection method.

[0037] In summary, this invention achieves synergistic optimization of garbage collection efficiency and system stability by constructing an adaptive GC strategy based on cluster capacity. Its core advantage lies in: dynamically sensing the cluster capacity status, employing conventional relocation and reclamation to ensure timely space reclamation during low capacity utilization, and switching to a relocation-free reclamation method during high capacity utilization, performing space punching release based on continuous garbage characteristics, thus achieving adaptive matching between the GC strategy and cluster capacity. This mechanism fundamentally eliminates the contention and consumption of network bandwidth and disk throughput caused by the two-stage I / O of read and write operations in the traditional relocation process, and avoids the continuous crowding of remaining capacity by incremental writes under high capacity levels, thereby effectively resolving the contradiction between garbage space release and front-end business write blocking. Ultimately, this invention significantly improves GC collection efficiency under high-pressure scenarios, maintains the dynamic stability of cluster capacity and system reliability, and ensures the continuity and quality of service of front-end business writes.

[0038] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0039] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this application will be briefly introduced below. It should be noted that the drawings only show some embodiments of the present invention. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the overall implementation process of the hard disk garbage collection method described in this application.

[0041] Figure 2 This is a structural diagram of the hard disk waste recycling device of this application.

[0042] Figure 3 This is a diagram of the cluster topology involved in the embodiments of this application.

[0043] Figure 4 This is a schematic diagram illustrating the storage object metadata and data association relationships in the embodiments of this application.

[0044] Figure 5 This is a flowchart illustrating the implementation of the adaptive GC strategy in the embodiments of this application.

[0045] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0046] Caption: Processor-310, Communication Interface-320, Memory-330, Communication Bus-340. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0048] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0049] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0050] Example: A method for hard drive garbage collection

[0051] The implementation process of this plan is as follows:

[0052] Assuming a 3-node cluster, with each node having 4 disks, the cluster topology is as follows: Figure 3 As shown.

[0053] like Figure 3The index pool is primarily used to store metadata information corresponding to user data, while the data pool is used to store the actual user data; user data (single value, i.e., sv or array value) The value, where array values ​​are generally described in a fixed size such as 512 bytes (e.g., [0,3], a total of 4 groups * 512 = 2K bytes), is first mapped to the index object during writing. The index object is then mapped to the corresponding master node. For example, the master node for index_obja1 / b1 / c1 is node0, the master node for index_obja2 / b2 / c2 is node1, and the master node for index_obja3 / b3 / c13 is node2. Then, on the master node, after aggregation writing, each storage object is requested. Assuming 3-replica redundancy, the requested storage objects are obj0 (tgt0, tgt4, tgt8), obj1 (tgt1, tgt5, tgt9), and obj2 (tgt2, tgt5, tgt9). That is, the write requests for a1 / b1 / c1 are aggregated together through obj0, the write requests for a2 / b2 / c2 are aggregated together through obj1, and the write requests for a3 / b3 / c3 are aggregated together through obj3. After aggregation, the metadata information of the Index object (such as the index container `cont`, the index obj object ` / dkey / akey / recx`, and the backend storage address) is recorded and stored in the metadata space of the index pool. Simultaneously, it is recorded in `obj0` to locate its storage position. When this object repeatedly overwrites new `obj` storage objects, the data in the old `obj` storage object becomes garbage data. When the amount of garbage in the storage object reaches the GC reclamation threshold (e.g., a static threshold of 80%), the garbage space can be reclaimed using the GC garbage collection algorithm. Specifically, the index metadata information recorded in the `obj` storage object is read and used for reverse lookup. The reverse lookup result, i.e., the metadata information of the index object (such as the index container `cont`, the index obj object ` / dkey / akey / recx`, and the storage object ID), is compared with the information stored on the storage object to determine whether the data is garbage or valid data. If it is valid data, it needs to be moved to a new storage object. Since data migration (reading from disk first and then writing to disk) involves reading and writing to disk, it is subject to bandwidth and concurrency limitations of network and hard disk resources, affecting garbage collection efficiency. Especially when hard disk space is insufficient, the writes caused by GC migration can cause the hard disk space to be filled up, blocking the release of garbage space and thus interrupting foreground writes. Based on this, this invention proposes an optimization method to improve GC collection efficiency based on adaptive adjustment of GC strategy according to cluster capacity. The specific implementation steps are as follows:

[0054] 1) Create garbage collection bins, with each bin containing 5 levels. A total of 20 bins will be created for garbage levels from 0 to 100, namely [0,5), [5,10), [10,15)...[95,100]. The garbage will be collected in order of decreasing garbage level, i.e., garbage will be collected first from [95,100], then [90,95)...[0,5).

[0055] 2) Objects to be recycled are placed into the appropriate bins: Using a standard garbage bitmap marking algorithm, the proportion of garbage data in the objects to be recycled is calculated, and the objects are placed into the corresponding garbage bins. For example, if the amount of objects to be recycled is 128M, and the smallest write unit for an object is 4K, with each 4K represented by 1 bit, then the 128M objects can be represented by 32K bits. 0 indicates valid data, and 1 indicates garbage. The covered parts of the object are marked as 1. By counting the number of 1s in the bitmap, the corresponding garbage rate can be calculated. If there are a total of 5K 1s in the bitmap, then the amount of garbage is 5K * 4K = 20M, and the garbage rate is 20M / 128M = 15.625%, then it is inserted into the bin in the range [15, 20).

[0056] 3) Dynamic switching of GC reclamation method: When the cluster capacity is low, space can be reclaimed through regular GC moving; when the cluster capacity is high, regular GC moving and reclamation may lead to the risk of the remaining disk space being filled and blocking I / O. At this time, it is switched to the non-moving method based on the continuous characteristics of garbage data statistics; for example, when the cluster capacity reaches 85%, switch to the non-moving GC method.

[0057] 4) Non-moving method based on continuous garbage data characteristics statistics: The total length of continuous garbage data is determined according to the metadata information and bitmap information recorded in the storage object. When the total length of garbage reaches the continuous garbage collection threshold, the entire garbage is punched to release the space it occupies. At the same time, the metadata is corrected to indicate that it has been punched. For example, if metadata meta1 indicates that the data is garbage, the corresponding data length is 4M, and the distribution position on the storage object is [1024, 2047], meta2 also indicates that the data is garbage, the corresponding data length is 2M, and the distribution position on the storage object is [2048, 2559], and meta3 also indicates that the data is garbage, the corresponding data length is 1M, and the distribution position on the storage object is [2560, 2815], then the length of continuous garbage is 4M + 2M + 1M = 7M. If the continuous garbage threshold is 4M, the GC condition is met, and the 7M data space corresponding to meta1, meta2, and meta3 can be punched as a whole to release 7M of data space. Figure 4 The image shows the metadata and data relationships of the stored objects.

[0058] 5) Adaptive adjustment method for continuous garbage collection threshold: To adapt to the redundancy strategy of both replicas and EC erasure coding, a baseline garbage collection threshold is set according to stripe alignment (ensuring that each shard can be released). The baseline garbage collection threshold is automatically adjusted according to the urgency of the capacity to release as much garbage space as possible and reduce the risk of space being written to the limit. For example, if the cluster capacity is used to 92%, and it is EC4+2 redundancy, the baseline garbage collection threshold is 1M, then the continuous garbage collection threshold is 4*1M=4M. That is, when the continuous garbage space reaches or exceeds 4M, it can be punched out as a whole. If it is replica redundancy (such as 3 replicas), the baseline garbage collection threshold can be used directly as the continuous garbage collection threshold. That is, when the continuous garbage space reaches or exceeds 1M, it can be punched out as a whole to release space.

[0059] Table 1 Adaptive Adjustment Method for Benchmark Recovery Threshold

[0060]

[0061] 6) Whether the metadata record data has been punched: If the data represented by the metadata has been punched, such as meta1, meta2, and meta3 in 4), the punched mark is recorded on it and removed from the disk for filtering during the next GC rescan process. That is, the space that has been punched will not be counted and released again, but will be skipped directly.

[0062] 7) Correction of garbage amount of storage objects: For storage objects that have been successfully punched, their actual garbage amount needs to be corrected; for example, if the initial garbage amount of a storage object is 20M, and 8M of space has been punched, then the garbage amount should be changed to 20-8=12M.

[0063] 8) Reswitching storage objects into different bins: Calculate the garbage rate based on the amount of garbage and capacity of the storage object after it has been punched, and reswitch bins according to the garbage rate. For example, if the storage object capacity is 128M and the initial amount of garbage is 20M, the garbage rate is 20M / 128M=15.625%, which is distributed in the bin [15,20). If 8M of space has been punched, the garbage rate becomes (20-8) / 128=9.375%, and it is then moved to the bin [5,10). The purpose is to reduce the probability of low-garbage storage objects being selected.

[0064] Figure 5 The diagram shows the implementation flow of the adaptive GC strategy in this invention, including:

[0065] (1) Start iterating from high garbage volume, determine whether the cluster capacity exceeds the set threshold such as 85%. If it does not reach the threshold, proceed with the normal migration logic. Check the metadata to see if it is garbage. If it is garbage, do not migrate. Otherwise, read out the old data first and then write it to the new storage object to realize the migration.

[0066] (2) When the capacity exceeds the 85% threshold, the non-moving mode is used. The continuous amount of garbage is counted according to the reverse lookup results. If it exceeds the threshold, the entire garbage can be punted, and the meta metadata punting information is marked and the garbage bin distribution is adjusted; otherwise, the statistics are cleared and the count is restarted.

[0067] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0068] like Figure 6 As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the aforementioned hard disk garbage collection method.

[0069] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0070] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types:

[0071] Object-oriented programming languages, such as Java, Smalltalk, C++, etc.

[0072] Conventional procedural programming languages, such as "C" or similar programming languages.

[0073] The execution methods of program code include, but are not limited to:

[0074] It runs entirely on the user's computer;

[0075] Part of it executes on the user's computer, and part of it executes on a remote computer;

[0076] Execute as a standalone software package;

[0077] It is executed entirely on a remote computer or server.

[0078] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0079] Furthermore, this application also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the various steps of the hard disk garbage collection method disclosed in this application.

[0080] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following:

[0081] (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0082] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0083] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0084] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0085] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0086] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0087] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0088] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0089] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0090] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0091] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0092] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program contains program code for executing the hard disk garbage collection method disclosed in this application. When the computer program is executed by a processing device, it can achieve the functions defined in the embodiments of this application.

[0093] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the above-disclosed concept.

[0094] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for hard drive garbage collection, characterized in that, The method is applied to a distributed storage system employing redirected write technology and includes the following steps: S1. Waste volume bin management: Divide the waste volume of stored objects into multiple ranges and create corresponding recycling bins, and perform recycling in order of priority from high to low waste volume; S2. Object binning and garbage quantity statistics: Calculate the proportion of garbage quantity of objects to be recycled and classify them into the corresponding recycling bins; S3, GC strategy dynamic switching: Monitor cluster capacity utilization. When the capacity utilization is lower than the preset threshold, perform GC collection based on data migration; when the capacity utilization reaches or exceeds the preset threshold, switch to a non-migration GC collection method based on continuous characteristics of garbage data. S4. Handling-free garbage collection: Based on the metadata and bitmap information of the stored object, identify continuous garbage data segments. When the total length of continuous garbage data reaches the continuous garbage collection threshold, perform a punching operation on the continuous garbage data segments to release space and mark the corresponding metadata as punched. S5. Adaptive adjustment of continuous garbage collection threshold: The continuous garbage collection threshold is adaptively adjusted according to the urgency of cluster capacity utilization. The threshold is set according to the stripe alignment principle of the redundancy strategy. S6. Waste Quantity Correction and Bin Switching: For storage objects with punched holes to release space, correct their waste quantity ratio and redistribute them to the corresponding recycling bins according to the corrected waste quantity ratio.

2. The method according to claim 1, characterized in that, In step S1, the waste volume ratio is divided into 5% increments, and a total of 20 recycling bins are set up, namely [0,5), [5,10), ..., [95,100].

3. The method according to claim 1, characterized in that, In step S2, the method for calculating the proportion of garbage data in the storage object to be recycled is as follows: using a bitmap marking algorithm, valid data and garbage data are marked at the smallest writing unit of the storage object, the number of garbage data marking bits in the bitmap is counted, and the proportion of garbage data is calculated.

4. The method according to claim 1, characterized in that, In step S3, the preset threshold is 85%; The GC collection based on data migration includes: determining the valid data in the storage object through reverse lookup of metadata, reading the valid data from the old storage object and writing it to the new storage object, and deleting the old storage object after the data migration is completed.

5. The method according to claim 1, characterized in that, In step S4, the method for identifying continuous garbage data segments includes: determining the position and length of each garbage data segment based on the metadata information of the stored object, determining whether adjacent garbage data segments are continuous, and merging them to calculate the total length if they are continuous.

6. The method according to claim 1, characterized in that, In step S5, the redundancy strategy includes replica redundancy and erasure coding redundancy; for the erasure coding redundancy strategy, the continuous garbage collection threshold is the product of the baseline garbage collection threshold and the number of stripe fragments; for the replica redundancy strategy, the continuous garbage collection threshold is equal to the baseline garbage collection threshold.

7. The method according to claim 6, characterized in that, The baseline reclamation threshold is adaptively adjusted based on the urgency of cluster capacity utilization, specifically as follows: When the capacity utilization rate is in the range of [85%, 90%), the baseline recovery threshold is 2M; When the capacity utilization rate is in the range of [90%, 95%), the baseline recovery threshold is 1M; When the capacity utilization rate is in the range of [95%, 100%], the baseline recovery threshold is 32K.

8. The method according to claim 1, characterized in that, In step S6, the method for correcting the garbage amount ratio is as follows: subtract the amount of space released by punching holes from the original garbage amount of the stored object to obtain the corrected garbage amount, and then calculate the corrected garbage amount ratio.

9. The method according to claim 1, characterized in that, The method further includes: during subsequent GC scanning, skipping the repeated counting and release of punched space based on the punched markers in the metadata.

10. A hard drive waste recycling device, characterized in that, The apparatus is applied to a distributed storage system employing redirected write technology, and during operation implements the steps of the hard disk garbage collection method as described in any one of claims 1-9, including: The garbage bin module is used to divide the garbage amount of stored objects into multiple ranges and create corresponding recycling bins, and perform recycling in order of priority from high to low garbage amount. The object binning module is used to calculate the percentage of garbage objects to be recycled and classify them into the corresponding recycling bins. The strategy switching module is used to monitor the cluster capacity utilization. When the capacity utilization is lower than the preset threshold, it triggers GC reclamation based on data migration; when the capacity utilization reaches or exceeds the preset threshold, it switches to a non-migration GC reclamation method based on continuous feature statistics of garbage data. The non-moving recycling module is used to identify continuous garbage data segments based on the metadata and bitmap information of the stored object. When the total length of the continuous garbage data reaches the continuous garbage recycling threshold, the module performs a punching operation on the continuous garbage data segments to release space and marks the corresponding metadata as punched. The threshold adjustment module is used to adaptively adjust the continuous garbage collection threshold according to the urgency of cluster capacity utilization. The threshold is set according to the stripe alignment principle of the redundancy strategy. The correction switching module is used to correct the garbage amount ratio of storage objects that have been punched to free up space, and then reallocate them to the corresponding recycling bins according to the corrected garbage amount ratio.

Images