A method, device and electronic equipment for improving hard disk garbage collection efficiency

Abstract

The application discloses a method and device for improving hard disk garbage collection efficiency and electronic equipment, which are applied to a distributed storage system using a ROW technology. In the GC process, the method reads index metadata information of a storage object to be collected, constructs an object tree according to a hierarchical relationship of an index container, an index object, a data key / attribute key / record interval, performs aggregated sorting and record interval merging and reorganization on valid index objects, performs batch flushing when GC concurrency reaches an upper limit or aggregation times out, and writes target storage objects according to a historical association relationship between the index objects and the storage objects, so that data of the same index object is preferentially stored in a centralized manner. The application improves garbage concentration of the storage object, effectively reduces data moving amount during GC in an overwrite writing scene, and significantly improves GC collection efficiency and maintains cluster capacity stability.

Description

Technical Field

[0001] This application relates to the field of hard disk waste recycling technology, specifically to a method, apparatus, and electronic device for improving hard disk waste recycling efficiency. Background Technology

[0002] To fully utilize hard drive write performance, the industry commonly employs Redirect-on-Write (ROW) technology, which aggregates data to be written to the hard drive pool into sequential large I / O operations before writing it to the allocated objects. In the field of distributed storage, append-only write mechanisms can significantly improve data write performance, so ROW technology also adopts this mechanism. However, appending the same data generates garbage data, so garbage collection (GC) algorithms are needed to reclaim invalid space. Since ROW typically uses a large object storage strategy (e.g., a single object of 128MB), the common GC implementation is to scan objects and identify those whose garbage data volume reaches a preset threshold, and then reclaim space by data migration (i.e., reading the valid data from the old object address and writing it to the new object, then deleting the old object to free up space).

[0003] During concurrent aggregated writes of ROW objects, as garbage collection (GC) continues, the data key (dkey) of the same foreground index object may be scattered across multiple different data pool objects. When a subsequent overwrite write is performed on the same object, it will lead to fragmentation of garbage data distribution within the object and a decrease in garbage concentration. This problem presents GC with a dilemma: either garbage data cannot be reclaimed in a timely manner, or a large amount of valid data needs to be moved during reclamation, resulting in low GC efficiency. Summary of the Invention

[0004] To address the aforementioned problems in existing technologies, this application proposes a novel method for improving hard disk garbage collection efficiency. More specifically, this application aims to solve the problem of optimizing garbage collection (GC) efficiency in distributed storage systems based on redirected write (ROW) technology by proposing a method for improving garbage concentration through object aggregation.

[0005] Specifically, this application provides the following technical solutions:

[0006] The first aspect of this application provides a method for improving hard disk garbage collection efficiency, the method being applied to a distributed storage system employing Redirect-on-Write (ROW) technology, such as... Figure 1 As shown, it includes the following steps:

[0007] S1. Constructing the object tree: During the garbage collection (GC) process, the index metadata information in the storage objects to be reclaimed is read, and the object tree is constructed according to the hierarchical relationship of index container, index object, data key / attribute key / record range, and the total amount of data of each index object is recorded;

[0008] S2. Object Tree Sorting and Aggregation: Valid index objects are inserted into the object tree for aggregation, and sorted in descending order of total data volume of the index objects; record intervals under the same attribute key are merged according to the principles of adjacent merging and cross-reorganization.

[0009] S3. Batch flushing of object tree: When the number of concurrent objects in garbage collection reaches the preset limit, or the object tree aggregation time exceeds the preset timeout threshold, the aggregated object tree is written to the storage object in batches; the unaggregated part waits for the next round of aggregation processing;

[0010] S4. Storage Object Selection and Writing: Based on the historical relationship between the index object and the storage object and the amount of data to be written, select the target storage object for writing to the index objects in batches. This will prioritize writing the data of the same index object to storage objects with historical relationships, thereby improving the garbage concentration of storage objects.

[0011] Furthermore, in the method of this application, the index metadata information mentioned in step S1 includes: index container identifier, index object identifier, data key, attribute key, record range, and backend storage address.

[0012] Furthermore, in the method of this application, the adjacent merging and cross-reorganization principles in step S2 include: merging record intervals with adjacent indexes under the same attribute key into continuous intervals; and reorganizing record intervals with index crosses into continuous intervals covering the index ranges of both parties.

[0013] Furthermore, in the method of this application, step S4, which involves selecting target storage objects for writing to index objects in batches based on the historical association between index objects and storage objects and the amount of data to be written, includes:

[0014] S41. Prioritize processing associated storage objects: Iterate in descending order of the amount of data in the index objects. If there is a storage object that has been written to the index object, and the remaining writable space of the storage object is not less than the amount of data to be written to the index object, then write directly. If the remaining writable space is insufficient, close the storage object after it is full and request a new storage object. At the same time, adjust the sorting position of the remaining data of the index object in the object tree according to the remaining data amount.

[0015] S42. Handling unassociated storage objects: If there are no historically associated storage objects, then empty storage objects are selected for writing first; if there are no empty storage objects, then the storage object with the smallest difference in data volume between the associated index object and the index object to be written is selected for writing; when there are multiple minimum differences, the storage object with the largest remaining writable space is selected first; if the remaining writable space is insufficient, the same write-full-close and sorting adjustment operations as in step S41 are performed.

[0016] Furthermore, in the method of this application, after applying for a new storage object, the new storage object is included in the current garbage collection concurrent storage object set to participate in subsequent iteration selection.

[0017] Furthermore, in the method of this application, the preset upper limit mentioned in step S3 is a threshold for the number of concurrent garbage collection objects, and the preset timeout threshold is the maximum waiting time for object tree aggregation.

[0018] Furthermore, in the scenario of overwrite, the method of this application causes the garbage data generated by the overwriting of the same index object to be aggregated on the storage object. When the storage object is selected for garbage collection, the amount of effective data moved is reduced.

[0019] A second aspect of this application provides an apparatus for improving hard disk garbage collection efficiency. The apparatus is applied to a distributed storage system employing redirected write technology and, during operation, implements the steps of the aforementioned method for improving hard disk garbage collection efficiency, such as... Figure 2 As shown, the device includes:

[0020] The object tree construction module is used during garbage collection to read the index metadata information in the storage objects to be collected, construct the object tree according to the hierarchical relationship of index container, index object, data key / attribute key / record range, and record the total amount of data of each index object;

[0021] The sorting and aggregation module is used to insert the determined valid index objects into the object tree for aggregation, and sort them in descending order of the total data volume of the index objects; and to merge record intervals under the same attribute key according to the principles of adjacent merging and cross recombination.

[0022] The batch flush module is used to write the aggregated object tree to the storage object in batches when the number of concurrent garbage collection objects reaches the preset limit or the object tree aggregation time exceeds the preset timeout threshold; the unaggregated part waits for the next round of aggregation processing.

[0023] The storage selection module is used to select target storage objects for writing to index objects in batches based on the historical relationship between index objects and storage objects and the amount of data to be written. This ensures that data of the same index object is preferentially written to storage objects with historical relationships, thereby improving the garbage concentration of storage objects.

[0024] Furthermore, in the device of this application, the storage selection module includes:

[0025] The associated processing unit is used to iterate in descending order of the amount of index object data. If there is a storage object that has been written to the index object and there is enough remaining space, it is written directly. If there is insufficient space, the storage object is closed after it is filled and a new storage object is requested. At the same time, the sorting position of the index object in the object tree is adjusted.

[0026] The unassociated processing unit is used to prioritize selecting an empty storage object when there is no historical associated storage object; if there is no empty storage object, it selects the storage object with the smallest difference in data volume between associated index objects, and if the difference is the same, it prioritizes the one with the largest remaining writable space; if the space is insufficient, it performs the same write-full shut-off and sorting adjustment operations as the associated processing unit.

[0027] A third aspect of this application provides an electronic device, including: a memory and a processor;

[0028] Memory: Used to store computer programs;

[0029] Processor: Used to execute the computer program to implement the steps of the aforementioned method for improving hard disk garbage collection efficiency.

[0030] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned method for improving hard disk garbage collection efficiency.

[0031] In summary, compared with the prior art, the advantages of this invention are as follows: This solution greatly improves the continuity of data writing on the disk by aggregating and sorting index objects and selecting compatible storage objects during the GC process. In the case of overwrite, the garbage generated by the same index object being overwritten is aggregated on the storage object, which increases the garbage concentration rate of the storage object. When the storage object is selected by GC for garbage collection, the amount of data migration can be effectively reduced, thereby improving GC collection efficiency and maintaining the stability of cluster capacity.

[0032] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0033] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this application will be briefly introduced below. It should be noted that the drawings only show some embodiments of the present invention. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating the overall implementation process of the method for improving hard drive junk file recycling efficiency in this application.

[0035] Figure 2 This is a structural diagram of the device for improving hard disk junk recycling efficiency according to this application.

[0036] Figure 3 This is a diagram of the cluster topology involved in the embodiments of this application (this example is a 3-node cluster, with 4 disks under each node).

[0037] Figure 4 This is a schematic diagram of data partitioning in this application.

[0038] Figure 5 This is a tree structure diagram of the index objects in this scheme.

[0039] Figure 6 This diagram illustrates the association and iteration method between the index object and the storage object in this scheme.

[0040] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0041] Caption: Processor-310, Communication Interface-320, Memory-330, Communication Bus-340. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0043] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0044] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0045] Example: A method to improve hard drive garbage collection efficiency

[0046] The implementation process of this plan is as follows:

[0047] Assuming a 3-node cluster, with each node having 4 disks, the cluster topology is as follows: Figure 3 As shown.

[0048] like Figure 3As shown, the index pool is mainly used to store metadata information corresponding to user data, while the data pool is used to store the actual user data; user data (single value, i.e., sv or array value) The value, where array values ​​are generally described in a fixed size such as 512 bytes (e.g., [0,3], a total of 4 groups * 512 = 2K bytes), is first mapped to the index object during writing. The index object is then mapped to the corresponding master. For example, index_objA1 / B1 / C1 is master node0, index_objA2 / B2 / C2 is master node1, and index_objA3 / B3 / C3 is master node2. Then, on the master, after aggregation writing, each storage object is requested. Assuming 3-replica redundancy, the requested storage objects are obj0 (tgt0, tgt4, tgt8), obj1 (tgt1, tgt5, tgt9), and obj2 (tgt2, tgt5, tgt9). That is, the write requests of A1 / B1 / C1 are aggregated together through obj0, the write requests of A2 / B2 / C2 are aggregated together through obj1, and the write requests of A3 / B3 / C3 are aggregated together through obj3. After aggregation, the metadata information of the index object (such as the index container `cont`, the index obj object ` / dkey / akey / recx`, and the backend storage address) is recorded and stored in the metadata space of the index pool. Simultaneously, it is recorded in `obj0` to locate its storage location. When this object repeatedly overwrites new `obj` storage objects, the data in the old `obj` storage object becomes garbage data. When the amount of garbage in the storage object reaches the GC reclamation threshold (e.g., a static threshold of 80%), the garbage space can be reclaimed using the GC garbage collection algorithm. Specifically, the index metadata information recorded in the `obj` storage object is read and used for reverse lookup. The reverse lookup result, i.e., the metadata information of the index object (such as the index container `cont`, the index obj object ` / dkey / akey / recx`, and the storage object ID), is compared with the information stored on the storage object to determine whether the data is garbage or valid data. If it is valid data, it needs to be moved to a new storage object.

[0049] To improve the concentration rate of garbage objects after relocation, this application proposes a method to improve GC collection efficiency based on object aggregation. The specific implementation steps are as follows:

[0050] 1. Construct an index object tree: such as Figure 5 As shown, an object tree is generated according to the index container, index object, and dkey / akey / recx.

[0051] 2. Object Tree Sorting and Aggregation: Index objects that are not garbage and whose data reading is complete are inserted into the object tree for aggregation. The total amount of data for each object is recorded, and the objects are sorted from largest to smallest. Recx objects under the same akey are sorted from largest to smallest, and intersecting or adjacent recx objects are recombined and merged. For example, if akey0 has recx0[0,8], recx1[9,15], and recx2[12,18], then according to the adjacent merging principle, recx0 and recx1 are merged into [0,15]; according to the intersecting recombining principle, [0,15] and recx2[12,18] are recombined into [0,18].

[0052] 3. Batch flushing of object trees: When the concurrency of GC objects reaches the limit or the object tree aggregation exceeds the aggregation time, the object tree is written in batches. For example, if the current number of concurrent GC objects is 10 and the preset object tree aggregation timeout is 100 seconds, the object tree will be flushed and written in batches when the data in the 10 GC objects has been processed or the object tree aggregation time has exceeded 100 seconds. The remaining unaggregated part will wait for the next round of aggregation.

[0053] 4. Selecting Storage Objects for Writing: When selecting backend storage objects for writing, adjustments need to be made based on the object's relationships and the amount of data being written. The relationship methods are as follows: Figure 6 As shown, it specifically includes:

[0054] 4.1 Handling previously written index objects: Iterate through index objects and concurrent storage objects in descending order. If an iterated storage object has previously written to the index object, write the index object to its associated storage object. If the index object's length is less than the storage object's writable length, write it directly; if the index object's length is greater than the storage object's writable length, close the storage object when it's full and allocate a new storage object. Simultaneously, adjust the index object's position in the index tree according to its remaining data length. For example, if the current batch flushes 20 index objects from iobj0 to iobj1... 9. The object storage concurrency is 10, namely dobj0~dobj9. When iterating to iobj0, it checks if there is a storage object associated with iobj0 among dobj0~dobj9. If it is found, such as dobj5, the data of iobj0 is directly written to dobj5. If the total data of iobj0 is 8M, but the space available for writing in dobj5 is only 5M, then the 5M data of iobj0 is written to dobj5. After dobj5 is written, it is closed and a new storage object such as dobj10 is requested. At the same time, the position of iobj0 in the index tree is adjusted according to the remaining 3M of data.

[0055] 4.2 Handling Unwritten Index Objects: Iterate through concurrent storage objects. If a new empty storage object exists, write it directly. Otherwise, find the index object associated with the storage object and the index object to be written to with the smallest difference. If multiple objects have the smallest difference, prioritize the one with the largest writable length. If the length of the index object is less than the writable length of the storage object, write it directly and record the association between the index object and the storage object. If the length of the index object is greater than the writable length of the storage object, close the storage object when it is full and allocate a new storage object. Simultaneously, adjust the index object's position in the index tree according to the remaining data length of the index object. If the current batch flush has 20 index objects from iobj0 to iobj19, and the object storage concurrency is 10 from dobj0 to dobj9, when iterating to iobj0, if no associated storage object is found, and dobj9 is a newly allocated empty storage object, then write it to dobj9; if no empty storage object exists in the current concurrent storage objects, then find the data object with the smallest object difference associated with the dobj storage object and write it. For example, dobj0 is associated with iobj1, and dobj1 is associated with iobj2. Because the difference between iobj0 and iobj1 is small (from the perspective of data distribution, they have higher affinity), dobj0 is chosen to be written.

[0056] 4.3 Repeat the processing strategies in 4.1 and 4.2 to complete the writing of all objects.

[0057] Figure 4 The diagram shows the data partitioning in this scheme. As shown in the figure, assuming that an index object can store 4G of data and a dkey can store 4M of data, then 4G / 4M=1024 dkeys are needed to fill an object. Each dkey corresponds to an akey, and each recx is 512 bytes. Therefore, 4M / 512=8192 recxes are needed to fill a dkey, which means that the corresponding recx index is [0,8191].

[0058] Figure 5 The diagram shown is the tree structure of the index object in this solution. As shown, this solution aggregates and sorts objects according to obj / dkey / akey, with the following sorting principles:

[0059] 1) The obj objects are sorted according to the total amount of data;

[0060] 2) Sort dkey / akey by index in ascending order;

[0061] 3) Recx under the same akey are sorted in ascending order and reorganized according to the principle of adjacent intersection.

[0062] Figure 6 The diagram illustrates the association and iteration method between the index object and the storage object in this scheme. As shown, in this scheme, when an index object is written to a storage object, its associated storage object is found and written according to the affinity principle (in the diagram, iobj0-iobj2 represent the index objects during iteration, and iobja-iobji represent the index objects associated with the storage objects). The affinity judgment principle is as follows:

[0063] 1) Prioritize writing to storage objects that have already been written;

[0064] 2) Select an empty storage object to write;

[0065] 3) Select the storage object with the smallest difference between the index object associated with the storage object and write it. When there are multiple storage objects, prioritize the storage object with the largest writable length.

[0066] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0067] like Figure 7 As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the method described above for improving hard disk garbage collection efficiency.

[0068] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0069] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types:

[0070] Object-oriented programming languages, such as Java, Smalltalk, C++, etc.

[0071] Conventional procedural programming languages, such as "C" or similar programming languages.

[0072] The execution methods of program code include, but are not limited to:

[0073] It runs entirely on the user's computer;

[0074] Part of it executes on the user's computer, and part of it executes on a remote computer;

[0075] Execute as a standalone software package;

[0076] It is executed entirely on a remote computer or server.

[0077] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0078] Furthermore, this application also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the steps of the method for improving hard disk garbage collection efficiency disclosed in this application.

[0079] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following:

[0080] (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0081] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0082] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0083] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0084] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0085] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0086] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0087] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0088] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0089] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0090] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0091] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. The computer program contains program code for performing the method disclosed in this application for improving hard disk garbage collection efficiency. When the computer program is executed by a processing device, it can achieve the functions defined in the embodiments of this application.

[0092] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0093] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for improving hard drive garbage collection efficiency, characterized in that, The method is applied to a distributed storage system employing redirected write technology and includes the following steps: S1. Constructing the object tree: During garbage collection, read the index metadata information in the storage objects to be reclaimed, construct the object tree according to the hierarchical relationship of index container, index object, data key / attribute key / record range, and record the total amount of data for each index object; S2. Object Tree Sorting and Aggregation: Valid index objects are inserted into the object tree for aggregation, and sorted in descending order of total data volume of the index objects; record intervals under the same attribute key are merged according to the principles of adjacent merging and cross-reorganization. S3. Batch flushing of object tree: When the number of concurrent objects in garbage collection reaches the preset limit, or the object tree aggregation time exceeds the preset timeout threshold, the aggregated object tree is written to the storage object in batches; the unaggregated part waits for the next round of aggregation processing; S4. Storage Object Selection and Writing: Based on the historical relationship between the index object and the storage object and the amount of data to be written, select the target storage object for writing to the index objects in batches. This will prioritize writing the data of the same index object to storage objects with historical relationships, thereby improving the garbage concentration of storage objects.

2. The method according to claim 1, characterized in that, The index metadata information mentioned in step S1 includes: index container identifier, index object identifier, data key, attribute key, record range, and backend storage address.

3. The method according to claim 1, characterized in that, The adjacent merging and cross-reorganization principles described in step S2 include: merging adjacent record intervals under the same attribute key into a continuous interval; and reorganizing record intervals with index crosses into a continuous interval covering the index ranges of both parties.

4. The method according to claim 1, characterized in that, Step S4, which involves selecting target storage objects for writing to index objects in batches based on the historical association between index objects and storage objects and the amount of data to be written, includes: S41. Prioritize processing associated storage objects: Iterate in descending order of the amount of data in the index objects. If there is a storage object that has been written to the index object, and the remaining writable space of the storage object is not less than the amount of data to be written to the index object, then write directly. If the remaining writable space is insufficient, close the storage object after it is full and request a new storage object. At the same time, adjust the sorting position of the remaining data of the index object in the object tree according to the remaining data amount. S42. Handling unassociated storage objects: If there are no historically associated storage objects, then empty storage objects are selected for writing first; if there are no empty storage objects, then the storage object with the smallest difference in data volume between the associated index object and the index object to be written is selected for writing; when there are multiple minimum differences, the storage object with the largest remaining writable space is selected first; if the remaining writable space is insufficient, the same write-full-close and sorting adjustment operations as in step S41 are performed.

5. The method according to claim 4, characterized in that, After a new storage object is requested, the new storage object is included in the current garbage collection concurrent storage object set and participates in subsequent iteration selection.

6. The method according to claim 1, characterized in that, The preset upper limit mentioned in step S3 is the threshold for the number of concurrent objects in garbage collection, and the preset timeout threshold is the maximum waiting time for object tree aggregation.

7. The method according to claim 1, characterized in that, In the scenario of overwrite, the method causes the garbage data generated by the overwriting of the same index object to be aggregated on the storage object. When the storage object is selected for garbage collection, the amount of effective data moved is reduced.

8. A device for improving the efficiency of hard drive garbage collection, characterized in that, The apparatus is applied to a distributed storage system employing redirected write technology, and during operation implements the steps of the method for improving hard disk garbage collection efficiency as described in any one of claims 1-7, wherein the apparatus comprises: The object tree construction module is used during garbage collection to read the index metadata information in the storage objects to be collected, construct the object tree according to the hierarchical relationship of index container, index object, data key / attribute key / record range, and record the total amount of data of each index object; The sorting and aggregation module is used to insert the determined valid index objects into the object tree for aggregation, and sort them in descending order of the total data volume of the index objects; and to merge record intervals under the same attribute key according to the principles of adjacent merging and cross recombination. The batch flush module is used to write the aggregated object tree to the storage object in batches when the number of concurrent garbage collection objects reaches the preset limit or the object tree aggregation time exceeds the preset timeout threshold; the unaggregated part waits for the next round of aggregation processing. The storage selection module is used to select target storage objects for writing to index objects in batches based on the historical relationship between index objects and storage objects and the amount of data to be written. This ensures that data of the same index object is preferentially written to storage objects with historical relationships, thereby improving the garbage concentration of storage objects.

9. The apparatus according to claim 8, characterized in that, The storage selection module includes: The associated processing unit is used to iterate in descending order of the amount of index object data. If there is a storage object that has been written to the index object and there is enough remaining space, it is written directly. If there is insufficient space, the storage object is closed after it is filled and a new storage object is requested. At the same time, the sorting position of the index object in the object tree is adjusted. The unassociated processing unit is used to prioritize selecting an empty storage object when there is no historical associated storage object; if there is no empty storage object, it selects the storage object with the smallest difference in data volume between associated index objects, and if the difference is the same, it prioritizes the one with the largest remaining writable space; if the space is insufficient, it performs the same write-full shut-off and sorting adjustment operations as the associated processing unit.

10. An electronic device, characterized in that, include: Memory and processor; Memory: Used to store computer programs; Processor: for executing the computer program to implement the steps of the method for improving hard disk garbage collection efficiency as described in any one of claims 1-7.

Images