Garbage collection method, device and equipment of distributed storage system and medium

Abstract

This application provides a garbage collection method, apparatus, device, and medium for a distributed storage system. In this application, a hot / cold label value is introduced to indicate the hotness or coldness of a ROW object, and the garbage collection efficiency of the ROW object is determined based on the hot / cold label value and the proportion of garbage data in the ROW object. Since a smaller hot / cold label value indicates a colder ROW object, and a smaller hot / cold label value and a larger proportion of garbage data result in greater garbage collection efficiency, the garbage collection process can prioritize the collection of "cold and highly garbage" ROW objects. This effectively avoids the problem in related technologies where high-garbage ROW objects are blindly collected based solely on the proportion of garbage data, leading to frequent migration of cold data. This effectively alleviates write amplification and resource waste caused by this, and improves the overall performance of the storage system.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to methods, apparatus, equipment and media for garbage collection in distributed storage systems. Background Technology

[0002] In distributed storage systems, garbage collection (GC) is a key mechanism for freeing up storage space and optimizing system resource utilization.

[0003] Current garbage collection (GC) methods typically prioritize collecting ROW objects with the highest proportion of garbage data, based on the percentage of garbage data in the Redirect On Write (ROW) objects. While this approach can quickly free up storage space, cold data (i.e., data that is not frequently accessed) is frequently moved during garbage collection, which not only wastes system resources but also significantly exacerbates write amplification, severely impacting the overall performance of the storage system. Summary of the Invention

[0004] In view of this, embodiments of this application provide a garbage collection method, apparatus, device, and medium for a distributed storage system.

[0005] This application provides a garbage collection method for a distributed storage system, the method comprising: When a garbage collection event is detected, the hot / cold flag value of the write-redirected ROW object in at least one bin chain matched by the garbage collection event is obtained; the hot / cold flag value is used to indicate the hot / cold degree of the ROW object; wherein, the smaller the hot / cold flag value, the colder the ROW object, and the larger the hot / cold flag value, the hotter the ROW object. The garbage collection efficiency of a ROW object is determined based on its hot / cold label value and the proportion of garbage data in that ROW object; the smaller the hot / cold label value and the larger the proportion of garbage data, the greater the garbage collection efficiency of the ROW object. ROW objects are recycled based on their garbage collection efficiency.

[0006] As an example, the garbage collection GC metadata structure of each ROW object records the cold / hot tag value of that ROW object; Obtaining the hot / cold flag values ​​of ROW objects in at least one bin chain matched by the garbage collection event includes: For any ROW object in the bin chain, obtain the hot / cold tag value of the ROW object from the GC metadata structure of the ROW object.

[0007] As an example, the hot / cold tag value for each ROW object is determined based on the last modification timestamp of all data slices that make up the ROW object.

[0008] As an example, after a partial data slice in any ROW object has been reclaimed, the method further includes: A new ROW object is generated based on the remaining data slices in the ROW object and the data slices to be added to the ROW object; and the hot / cold flag value of the new ROW object is determined based on the last modification timestamp of the existing data slices in the new ROW object.

[0009] As an example, determining the garbage recycling benefit of a ROW object based on its hot / cold tag value and the proportion of garbage data in that ROW object includes: Obtain the percentage of valid data in the ROW object; the sum of the percentage of valid data and the percentage of garbage data in the ROW object is 1. Based on the hot / cold flag value of the ROW object, the creation time of the storage pool, and the current time, a normalized value for the hot / cold index of the ROW object is determined; the normalized value for the hot / cold index is negatively correlated with the hot / cold flag value. Based on the proportion of waste data, the proportion of valid data, and the normalized value of the hot / cold index of the ROW object, the waste recycling benefit of the ROW object is calculated; wherein, the proportion of waste data is positively correlated with the waste recycling benefit, the proportion of valid data is negatively correlated with the waste recycling benefit, and the normalized value of the hot / cold index is positively correlated with the waste recycling benefit.

[0010] As one embodiment, the process of recycling ROW objects based on their garbage collection efficiency includes: After obtaining the N bin chains that match the garbage collection event, a ROW object is selected from each bin chain in a preset order to obtain N ROW objects; where N is a positive integer. Select the ROW object with the highest garbage collection efficiency from the N ROW objects for recycling.

[0011] As an example, obtaining the cold / hot flag value of the write-time redirected ROW object in at least one bin chain matched by the garbage collection event is performed when the utilization rate of the storage pool is less than or equal to a set utilization rate threshold; When the utilization rate of the storage pool exceeds the utilization rate threshold, the method further includes: According to the corresponding recycling level from high to low, traverse at least one bin chain matched by the garbage collection event in turn; the higher the recycling level of any bin chain, the greater the proportion of garbage data of ROW objects on that bin chain. For the currently traversed bin chain, reclaim the ROW objects in the bin chain in the set order.

[0012] This application embodiment also provides a garbage collection device for a distributed storage system, the device comprising: The module is used to obtain the hot / cold flag value of the write-redirected ROW object in at least one bin chain matched by the garbage collection event when a garbage collection event is detected; the hot / cold flag value is used to indicate the hot / cold degree of the ROW object; wherein, the smaller the hot / cold flag value, the colder the ROW object, and the larger the hot / cold flag value, the hotter the ROW object. The determination module is used to determine the garbage collection efficiency of a ROW object based on its hot / cold label value and the proportion of garbage data in that ROW object; wherein, the smaller the hot / cold label value and the larger the proportion of garbage data, the greater the garbage collection efficiency of the ROW object. The recycling module is used to recycle ROW objects based on their garbage collection efficiency.

[0013] As an example, the garbage collection GC metadata structure of each ROW object records the cold / hot tag value of that ROW object; When the obtaining module performs the process of redirecting the cold / hot flag value of the ROW object during write operations in at least one bin chain matched by the garbage collection event, it is further configured to: For any ROW object in the bin chain, obtain the hot / cold tag value of the ROW object from the GC metadata structure of the ROW object.

[0014] As an example, the hot / cold tag value for each ROW object is determined based on the last modification timestamp of all data slices that make up the ROW object.

[0015] As an example, after a partial data slice in any ROW object is reclaimed, the reclamation module is further used for: The device further includes: an aggregation module; the aggregation module is used for: After a portion of the data slices in any ROW object are recycled, a new ROW object is generated based on the remaining data slices in the ROW object and the data slices to be added to the ROW object; and the hot / cold flag value of the new ROW object is determined according to the last modification timestamp of the existing data slices in the new ROW object.

[0016] As an example, when determining the garbage collection efficiency of a ROW object based on its hot / cold tag value and the proportion of garbage data, the determining module is further configured to: Obtain the percentage of valid data in the ROW object; the sum of the percentage of valid data and the percentage of garbage data in the ROW object is 1. Based on the hot / cold flag value of the ROW object, the creation time of the storage pool, and the current time, a normalized value for the hot / cold index of the ROW object is determined; the normalized value for the hot / cold index is negatively correlated with the hot / cold flag value. Based on the proportion of waste data, the proportion of valid data, and the normalized value of the hot / cold index of the ROW object, the waste recycling benefit of the ROW object is calculated; wherein, the proportion of waste data is positively correlated with the waste recycling benefit, the proportion of valid data is negatively correlated with the waste recycling benefit, and the normalized value of the hot / cold index is positively correlated with the waste recycling benefit.

[0017] As an example, when the recycling module performs the recycling of ROW objects based on the garbage collection efficiency of ROW objects, it is further configured to: After obtaining the N bin chains that match the garbage collection event, a ROW object is selected from each bin chain in a preset order to obtain N ROW objects; where N is a positive integer. Select the ROW object with the highest garbage collection efficiency from the N ROW objects for recycling.

[0018] As an example, obtaining the cold / hot flag value of the write-time redirected ROW object in at least one bin chain matched by the garbage collection event is performed when the utilization rate of the storage pool is less than or equal to a set utilization rate threshold; When the utilization rate of the storage pool exceeds the utilization rate threshold, the recycling module is further configured to: According to the corresponding recycling level from high to low, traverse at least one bin chain matched by the garbage collection event in turn; the higher the recycling level of any bin chain, the greater the proportion of garbage data of ROW objects on that bin chain. For the currently traversed bin chain, reclaim the ROW objects in the bin chain in the set order.

[0019] This application also provides an electronic device, including: a processor and a machine-readable storage medium for storing machine-executable instructions, wherein the machine-executable instructions, when run by the machine-readable storage medium, cause the processor to perform the steps of the above method.

[0020] This application also provides a machine-readable storage medium storing machine-executable instructions that, when executed, enable the implementation of the steps described above.

[0021] As can be seen from the above technical solutions, in this embodiment, a hot / cold label value is introduced to indicate the hotness or coldness of a ROW object, and the garbage collection efficiency of the ROW object is determined based on the hot / cold label value and the proportion of garbage data in the ROW object. Since a smaller hot / cold label value indicates a colder ROW object, and a smaller hot / cold label value and a larger proportion of garbage data result in a greater garbage collection efficiency, the garbage collection process can prioritize the collection of "cold and highly garbage" ROW objects. This effectively avoids the problem in related technologies where high-garbage ROW objects are blindly collected based solely on the proportion of garbage data, leading to frequent migration of cold data. This effectively alleviates the write amplification and resource waste caused by this, and improves the overall performance of the storage system. Attached Figure Description

[0022] Figure 1 A flowchart illustrating the method provided in the embodiments of this application; Figure 2 A flowchart illustrating the process of determining the benefits of waste recycling, provided for an embodiment of this application; Figure 3 Another flowchart illustrating the method provided in this application embodiment; Figure 4 This is a schematic diagram of the device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0024] Before introducing the method provided in the embodiments of this application, the existing technical problems will be explained: Garbage collection (GC) is a mechanism that automatically identifies and reclaims redundant, invalid, or expired data that is no longer needed by the system or users. Its core objective is to free up storage space, optimize resource utilization, and ensure data consistency and system performance. Currently, distributed storage systems typically prioritize garbage collection of ROW (Redirect-on-Write) objects, which have the highest proportion of garbage data, based on their percentage of garbage data.

[0025] However, this GC strategy leads to frequent movement of cold data during garbage collection. Cold data refers to data that is modified infrequently in the storage medium, while hot data refers to data that is modified frequently. When a ROW object with a high proportion of garbage data is selected during garbage collection, all still valid data slices within that ROW object must be read and written to a new ROW object to release the entire physical space occupied by the original ROW object. If these valid data slices happen to be cold data that has not been modified for a long time (such as historical data referenced by snapshots, which can also be called cold data slices), they will be moved from the old ROW object to the new ROW object. As the system continues to run, the new ROW object may accumulate garbage again due to subsequent write or overwrite operations, thus being selected for collection again in the future, causing these cold data slices to be moved again. This cycle repeats, and the same cold data slice may be repeatedly migrated, causing unnecessary write amplification, which not only wastes system I / O and computing resources, but also accelerates the wear and tear of the storage medium, seriously affecting the overall performance and lifespan of the storage system.

[0026] It should be noted that the hotness or coldness of any data slice is determined by the frequency of modification (overwrite, deletion, etc.) of that data slice, not by the frequency of reading or moving it. Do not confuse the two.

[0027] Based on this, embodiments of this application provide a large model optimization method, apparatus, electronic device, and medium to solve the problems of resource waste and write amplification caused by frequent handling of cold data during the garbage collection process.

[0028] See Figure 1 , Figure 1 This is a flowchart illustrating the method provided in an embodiment of this application. Optionally, this method can be applied to any node in a distributed storage system. All nodes share a storage pool, and the capacity of the storage pool is known when reclaiming each DPE. like Figure 1 As shown, the method may include the following steps: S101, when a garbage collection event is detected, obtain the hot / cold flag value of the write-redirected ROW object in at least one bin chain matched by the garbage collection event; the hot / cold flag value is used to indicate the hot / cold degree of the ROW object; wherein, the smaller the hot / cold flag value, the colder the ROW object, and the larger the hot / cold flag value, the hotter the ROW object.

[0029] In this embodiment, the system divides ROW objects into multiple recycling tiers based on the proportion of garbage data. There is a positive correlation between the recycling tier and the proportion of garbage data; the higher the recycling tier, the higher the proportion of garbage data in the corresponding ROW object. The specific mapping relationship between recycling tiers and the proportion of garbage data will be illustrated later and will not be elaborated here. For ease of description, in this embodiment, the recycling tier is represented by the "bin" field.

[0030] To efficiently manage ROW objects under each recycling tier, the system maintains a doubly linked list for each recycling tier to hold all ROW objects belonging to that tier. This linked list corresponding to a recycling tier is called a "bin chain". In other words, a bin chain refers to a chain-like data structure formed after ROW objects are categorized according to their proportion of garbage data.

[0031] Specifically, each bin chain consists of multiple nodes, each node corresponding to a ROW object and storing reference information pointing to that ROW object and its GC metadata. When a ROW object is generated (including initial creation, overwrite triggering, or garbage collection reorganization), the system determines the collection level to which the ROW object belongs based on its garbage data ratio, and inserts the node corresponding to the ROW object into the tail of the corresponding bin chain. When the ROW object is completely collected, the node corresponding to the ROW object is removed from the original bin chain.

[0032] In this embodiment, a garbage collection event refers to a trigger point determined by the storage system, either actively or passively, that requires initiating a garbage collection operation. The triggering mechanism for a garbage collection event can be based on various preset strategies, including but not limited to any one or a combination of the following: Periodic timer expiration: The system sets a background garbage collection timer (e.g., every 30 minutes). When the timer expires, the garbage collection event is automatically triggered to periodically optimize storage space.

[0033] Receive garbage collection instructions from external sources: respond to garbage collection instructions from administrators, operations and maintenance platforms, or upper-layer applications.

[0034] Detected a large number of deletion events: When the system detects a large number of data deletions, snapshot releases, or volume shrinking operations within a preset time period, it will proactively trigger garbage collection events to reclaim storage space.

[0035] Storage pool utilization sudden change: If the current utilization rate of the storage pool suddenly exceeds the first threshold (e.g., 85%) after the system is running normally and no garbage collection has been performed for a preset time (e.g., 1 hour), an emergency garbage collection event will be triggered immediately.

[0036] In this embodiment, the specific method for determining the hot / cold label value of any ROW object will be described later with specific embodiments, and will not be repeated here. The specific implementation method of the above step S101 will also be described in detail later, and will not be repeated here.

[0037] S102, Based on the hot and cold label values ​​of the ROW object and the proportion of garbage data of the ROW object, determine the garbage collection efficiency of the ROW object; wherein, the smaller the hot and cold label value and the larger the proportion of garbage data, the greater the garbage collection efficiency of the ROW object.

[0038] The specific implementation methods of the above steps S102 and S103 will be explained in detail later, and will not be repeated here.

[0039] S103, recycle ROW objects based on their garbage collection efficiency.

[0040] This concludes the process. Figure 1 The process is shown below.

[0041] pass Figure 1 As shown in the flowchart, in this embodiment, a hot / cold flag value is introduced to indicate the hotness or coldness of a ROW object. The garbage collection efficiency of the ROW object is determined based on the hot / cold flag value and the proportion of garbage data in the ROW object. Since a larger hot / cold flag value indicates a colder ROW object, and a larger hot / cold flag value and a larger proportion of garbage data result in a greater garbage collection efficiency, the garbage collection process can prioritize the collection of "cold and highly garbage" ROW objects. This effectively avoids the problem in related technologies where high-garbage ROW objects are blindly collected based solely on the proportion of garbage data, leading to frequent migration of cold data. This effectively alleviates the write amplification and resource waste caused by this, and improves the overall performance of the storage system.

[0042] The following explains the specific method for determining the hot / cold label value of any ROW object: As one example, the hot / cold tag value of any ROW object is determined based on the last modification timestamps of all data slices that make up the ROW object. The last modification timestamp of any data slice is the timestamp when the corresponding data slice was first written to or overwritten, and it is not updated during subsequent garbage collection processes due to the movement of data slices.

[0043] In this embodiment, ROW objects are managed using a pre-allocation method. That is, the system pre-allocates empty ROW storage space (i.e., ROW objects). At this time, the ROW objects do not yet contain any valid data slices, and their hot / cold flag values ​​are initialized to default values ​​(e.g., all zeros or a very large timestamp). This default value is only used as a placeholder and will be overwritten after subsequent data is filled in. It does not participate in the actual garbage collection benefit calculation.

[0044] After all data slices of the ROW object have been fully populated, a hot / cold flag value for the ROW object is determined based on the last modification timestamps of all the data slices that make up the ROW object. For example, the average of the last modification timestamps of all data slices is calculated, and this average value is used as the hot / cold flag value for the ROW object. This hot / cold flag value is used to characterize the overall hotness / coldness of the ROW object; the smaller the value, the colder the overall object (the longer the data has not been accessed or modified), and the larger the value, the hotter the overall object (the more frequently the data has been accessed or modified).

[0045] The last modification timestamp of the above data slice has the following meaning depending on its source: For a data slice being written for the first time, its last modification timestamp is the timestamp when the slice was first written.

[0046] For data slices generated by overwrite operations, their last modification timestamp is the timestamp when the overwrite operation occurred.

[0047] For data slices moved from other ROW objects through garbage collection, their last modified timestamp remains the original value before the move, that is, the original timestamp determined when the slice was first written or overwritten, rather than the timestamp of the move operation.

[0048] Based on the source combination of the data slices for each ROW object, the determination of the hot and cold label values ​​of the ROW object can be divided into the following three typical scenarios: Scenario 1: All data slices are being written for the first time: All data slices of this ROW object are first-write slices, and their hot / cold flag values ​​are determined directly based on the average of the first-write timestamps of all existing data slices.

[0049] Scenario 2: Overwrite triggers new ROW generation: Under the ROW mechanism, an overwrite operation does not directly modify data on the original ROW object. Instead, it allocates a new ROW object to hold the overwrite data. This new ROW object may contain: a new data slice generated by the overwrite operation, and a data slice to be added to the new ROW object. The data slice to be added to the ROW includes at least one of the following: the data slice written initially, data slices generated by other overwrite operations, or valid data slices reclaimed and moved from other ROW objects. At this point, the new ROW object recalculates the hot / cold flag value based on the latest modification timestamps of all its internal data slices.

[0050] Scenario 3: Garbage Collection Triggers New ROW Generation: During garbage collection, after any ROW object is reclaimed, some data slices within that ROW object are reclaimed, while others remain. Based on the remaining data slices in the original ROW object and the data slices to be added to it, a new ROW object is generated. The data slices to be added to the new ROW object include at least one of the following: data slices written for the first time, data slices generated by other overwrite writes, or valid data slices reclaimed and moved from other ROW objects. The new ROW object recalculates its hot / cold flag value based on the latest modification timestamps of all its internal data slices.

[0051] Using the above methods, the hot and cold tag values ​​can accurately reflect the overall latest modification timestamp distribution of all data slices within each ROW object, providing a reliable basis for subsequent calculations of recycling benefits based on hot / cold status and waste proportion.

[0052] The above provides a detailed explanation of the specific methods for determining the hot and cold tag values ​​of any ROW object.

[0053] The specific implementation method of step S101 above is described below: First, we will elaborate on at least one bin chain that matches the garbage collection event: The correspondence between each recycling level and the proportion of waste data is shown in Table 1.

[0054] Table 1 - Mapping Table of Recycling Tiers and Waste Data Percentage Based on this, the storage system is pre-configured with a mapping relationship between storage pool utilization and corresponding recycling tiers. This mapping relationship can be represented as a recycling tier table, as shown in Table 2. Table 2 defines the starting recycling tiers corresponding to different storage pool utilization ranges. During actual recycling, the system will start from the starting recycling tier and scan the bin chains corresponding to the starting recycling tier and all higher recycling tiers (i.e., recycling tiers with higher garbage percentages).

[0055] Table 2 - Mapping Table of Storage Pool Utilization and Initial Recycling Tier to be Scanned When a garbage collection event is triggered, the current storage pool utilization rate is first obtained. Then, based on Table 2, the starting collection level for this garbage collection is determined. The bin chains corresponding to the starting collection level and all higher collection levels above it are obtained; these bin chains are the bin chains matched for this garbage collection event. Subsequently, ROW objects will be selected from these bin chains for garbage collection benefit evaluation.

[0056] For example, if the current storage pool utilization rate is 55%, according to Table 2, the bin chains matched for this garbage collection event are bin chains 4 to 9. If the current storage pool utilization rate is 82%, according to Table 2, the bin chains matched for this garbage collection event are bin chains 1 to 9. This implementation can dynamically adjust the collection range according to the system load, providing a basis for subsequent benefit selection based on hot and cold tag values.

[0057] The specific implementation method for obtaining the cold / hot flag values ​​of ROW objects that are redirected on write in at least one bin chain matched by the garbage collection event will be further explained: The GC metadata structure of any ROW object records the cold and hot marking values ​​of that ROW object. The GC metadata structure in related technologies usually includes information such as maximum garbage size, tail identifier, reserve field, bin chain to which it belongs, and number of garbage collection failures. In this embodiment, N consecutive bytes are allocated from the reserve field of the existing structure to store the above-mentioned cold and hot marking values.

[0058] For example, 4 bytes are allocated from the 8-byte reserved field in the GC metadata structure (garbage_total_entry) to add a new age field (4 bytes), which stores a hot / cold flag value (a 32-bit timestamp) representing the hotness or coldness of ROW objects. The hot / cold flag value can be denoted as the AGE value.

[0059] struct garbage_total_entry { uint32_t garbage_size : 23; / / Maximum garbage size 4MB (stored in bytes) uint32_t tail : 1; / / Tail identifier (1 indicates the presence of a tail) uint32_t reserve : 8; / / Reserved field uint32_t age; / / AGE value uint8_t bin : 4; / / Bin chain to which it belongs (0-15) uint8_t coll_err : 4; / / Number of failed garbage collection attempts (skip if the limit is exceeded) By utilizing reserved bytes, the introduction of indicators of the hotness or coldness of ROW objects can be achieved without adding extra metadata storage overhead, ensuring good compatibility with the GC metadata structure of existing distributed storage systems.

[0060] Accordingly, the specific implementation of obtaining the cold and hot tag values ​​of ROW objects in at least one bin chain matched by the garbage collection event in step S101 above is as follows: for any ROW object in the bin chain, obtain the cold and hot tag value of the ROW object from the GC metadata structure of the ROW object.

[0061] The specific implementation method of step S101 above has been described in detail above.

[0062] The following section elaborates on the specific implementation method of determining the garbage collection efficiency of a ROW object based on its hot / cold label value and the proportion of garbage data in that ROW object: See Figure 2 , Figure 2 This is a schematic diagram illustrating the process for determining the benefits of waste recycling, provided as an embodiment of this application. Figure 2 As shown, the process may include the following steps: S201, obtain the effective data percentage of the ROW object.

[0063] In this embodiment, the sum of the effective data ratio and the garbage data ratio of the ROW object is 1.

[0064] S202, based on the cold / hot flag value of the ROW object, the creation time of the storage pool, and the current time, determine the cold / hot normalization value of the ROW object.

[0065] In this embodiment, the normalized value of hotness and coldness is negatively correlated with the value of hotness and coldness labeling.

[0066] For example, Tmp_AGE can be calculated using the following formula: in, This is the normalized value for hotness and coldness, which is negatively correlated with the hotness and coldness label value.

[0067] This is the current timestamp; This is the creation timestamp of the storage pool.

[0068] S203. Calculate the waste recycling efficiency of the ROW object based on the proportion of waste data, the proportion of valid data, and the normalized value of the ROW object's hot / coldness.

[0069] In this embodiment, the proportion of waste data is positively correlated with waste recycling efficiency. The proportion of valid data is negatively correlated with waste recycling efficiency. A higher proportion of waste data necessarily means a lower proportion of valid data, resulting in greater waste recycling efficiency; conversely, a lower proportion of waste data necessarily means a higher proportion of valid data, resulting in lower waste recycling efficiency.

[0070] The normalized value of heat and cold is positively correlated with the efficiency of waste recycling; the larger the normalized value, the greater the efficiency of waste recycling, and the smaller the normalized value, the smaller the efficiency of waste recycling. For example, the benefits of waste recycling can be calculated using the following formula: in, For the benefit of waste recycling; : Percentage of valid data (0-100, calculated based on the percentage of valid data slices within the ROW object); 100-u: Percentage of junk data; This is the normalized value for hotness and coldness, which is negatively correlated with the hotness and coldness label value.

[0071] In this embodiment, an algorithm is proposed that selects objects for recycling by combining the cold / hot flag value indicating the coldness of ROW objects with the proportion of garbage data. This algorithm is called the Garbage Collection Efficiency Score (GCES) algorithm (the recycling strategy based on this algorithm is called the high-efficiency recycling strategy). This algorithm tilts the recycling priority towards cold objects with a lot of garbage, thereby effectively avoiding the write amplification problem caused by frequent movement of cold data, reducing system overhead, and extending the life of storage media.

[0072] The above provides a detailed explanation of the specific implementation method for determining the waste recycling efficiency of a ROW object based on its hot / cold label value and the proportion of waste data in that ROW object.

[0073] The following section elaborates on the garbage collection efficiency of ROW objects: As an example, in accordance with Figure 2 The steps shown in the embodiment involve obtaining N bin chains matching the garbage collection event, and then selecting one ROW object from each bin chain in a preset order to obtain N ROW objects. Optionally, one ROW object can be selected from the head of each bin chain (the ROW objects in each bin chain are arranged in order of their entry time, with later entry times placing them further down the bin chain), resulting in N ROW objects. By selecting from the head of the chain, it can be ensured that the ROW objects of each matching bin chain have a fair chance to be included in the final collection scope, avoiding them being ignored due to prolonged stays at the tail of the chain. Of course, in other embodiments, selection can also be made from the middle or tail of the chain according to actual needs; this application is not specifically limited to this.

[0074] After obtaining these N ROW objects, according to Figure 2 The garbage collection efficiency of each ROW object is obtained in the manner shown. Then, these N garbage collection efficiencies are compared, and the ROW object with the highest garbage collection efficiency is selected as the ROW object to be collected this time, and the actual garbage collection operation is performed.

[0075] It should be noted that after each garbage collection cycle, the collected ROW objects are removed from their respective bin chains. Then, the process returns to the beginning of each bin chain, selecting one ROW object from the head of each bin, until a preset stop-collection condition is met.

[0076] By recycling ROW objects in the above manner, the system can select the ROW objects with the highest garbage collection efficiency in each recycling operation, thereby maximizing the overall recycling efficiency.

[0077] The above provides a detailed explanation of how to recycle ROW objects based on their garbage collection efficiency.

[0078] The aforementioned efficient recycling strategy is executed when the system's storage pool utilization rate is less than or equal to a set utilization threshold. When the storage pool utilization rate exceeds the utilization threshold, the method further includes: prioritizing the selection of ROW objects for recycling from the bin chain with the highest proportion of garbage data.

[0079] Specifically, when a waste recycling event is detected, if the system's storage pool utilization rate is less than or equal to a set utilization threshold (e.g., 80%), the aforementioned high-efficiency recycling strategy is executed; that is, the above-mentioned high-efficiency recycling strategy is implemented. Figure 1 The steps of the illustrated embodiment.

[0080] If the storage pool utilization rate exceeds a set utilization threshold, ROW objects are selected from the bin chain with the highest proportion of garbage data for recycling. This recycling method is called a greedy recycling strategy. That is, based on the storage pool utilization rate, M bin chains matching the garbage collection event are determined. These M bin chains are traversed sequentially from the highest recycling tier to the lowest recycling tier. For each traversed bin chain, the ROW objects attached to that bin chain are recycled sequentially from the head to the tail.

[0081] Thus, when the storage pool utilization rate is less than or equal to the set utilization rate threshold, the GCES algorithm is used to select objects for recycling, taking into account the combined effect of the proportion of effective data and the hotness of data, thereby reducing write amplification and waste of system resources. When the storage pool utilization rate is higher than the set utilization rate threshold, the traditional greedy algorithm is used to prioritize the recycling of ROW objects with a high proportion of garbage data, thereby maximizing the release of storage space and avoiding full write.

[0082] Here, it's important to clarify the specific implementation for determining whether a selected ROW object can be reclaimed: It checks whether the ROW object is no longer referenced, whether the ROW object has been completely flushed from the cache, and whether the log entry for the ROW object in the Write-Ahead Logging (WAL) has been deleted. If the ROW object is no longer referenced, the ROW object has been completely flushed from the cache, and the log entry for the ROW object has been deleted from the WAL, then the ROW object can be reclaimed. Otherwise, the ROW object is skipped, and the next ROW object is selected for reclamation.

[0083] In this embodiment, when the storage pool utilization rate is less than or equal to a set utilization threshold, the aforementioned efficient reclamation strategy is adopted to optimize the long-term performance of the system. When the storage pool utilization rate is higher than the set utilization threshold, the system automatically switches to a traditional greedy reclamation strategy, prioritizing the release of ROW objects with the largest amount of garbage to quickly reclaim space. This method of dynamically selecting a reclamation strategy based on the storage pool utilization rate ensures that the system always has efficient space reclamation capabilities under different load conditions, optimizes resource utilization during normal operation, and significantly improves the system's self-regulation capabilities.

[0084] To illustrate the method provided in this application in more detail, the following will be combined with... Figure 3 The solution provided in this application will be described in more detail by way of specific embodiments.

[0085] like Figure 3 As shown, the process may include the following steps: S301: Upon detecting a garbage collection event, determine whether the current utilization rate of the system's storage pool exceeds a set utilization threshold. If the result of step S301 is yes, then a greedy garbage collection strategy is initiated to reclaim ROW (i.e., steps S302 and S303 below are executed). If the result of step S301 is no, then a high-efficiency garbage collection strategy is initiated to reclaim ROW (i.e., steps S302 and S303 below are executed).

[0086] Specifically, initiating the greedy recycling strategy to recycle ROW includes the following steps S302 and S303: S302, obtain the current storage pool utilization rate, find the starting recycling level that matches the current storage pool utilization rate from the storage pool utilization rate and starting recycling level mapping table, and obtain the bin chain corresponding to the starting recycling level and all higher recycling levels above the starting recycling level. These bin chains are the bin chains matched for this garbage collection event.

[0087] Once M bin chains matching the garbage collection event are identified, S303 is executed, and these M bin chains are traversed sequentially from the highest collection level to the lowest collection level. For each bin chain traversed, the ROW objects attached to that bin chain are collected sequentially from the head to the tail.

[0088] Initiating a high-efficiency recycling strategy to recycle ROW specifically includes the following steps S304 and S303: S304. Obtain the current storage pool utilization rate. From the storage pool utilization rate and starting reclamation level mapping table, find the starting reclamation level that matches the current storage pool utilization rate. Obtain the bin chains corresponding to the starting reclamation level and all higher reclamation levels above the starting reclamation level. These bin chains are the bin chains matched for this garbage collection event.

[0089] After determining the N bin chains that match the garbage collection event, execute S305 to select a ROW object from the head of each bin chain, resulting in N candidate ROW objects.

[0090] S306: Select the ROW object with the highest garbage collection efficiency from N candidate ROW objects as the target ROW object, take the bin chain to which the target ROW object belongs as the target bin chain, and reclaim the target ROW object.

[0091] S307, delete the target ROW object on the target bin chain, and the ROW object after the target ROW object is placed at the head of the target bin chain again.

[0092] S308: Does the preset stop recycling condition meet? If the execution result of S308 is yes, the process ends; if the execution result of S308 is no, the process returns to execute S305.

[0093] The methods provided in the embodiments of this application have been described above. The apparatus provided in the embodiments of this application is described below: See Figure 4 , Figure 4 This is a structural diagram of the device provided in an embodiment of this application. Figure 4 As shown, the device 400 includes: an acquisition module 401, a determination module 402, and a recovery module 403.

[0094] The module 401 is used to obtain the hot / cold flag value of the write-redirected ROW object in at least one bin chain matched by the garbage collection event when a garbage collection event is detected; the hot / cold flag value is used to indicate the hot / cold degree of the ROW object; wherein, the smaller the hot / cold flag value, the colder the ROW object, and the larger the hot / cold flag value, the hotter the ROW object. The determination module 402 is used to determine the garbage collection efficiency of a ROW object based on the cold and hot label values ​​of the ROW object and the proportion of garbage data of the ROW object; wherein, the smaller the cold and hot label values ​​and the larger the proportion of garbage data, the greater the garbage collection efficiency of the ROW object. The recycling module 403 is used to recycle ROW objects based on their garbage collection efficiency.

[0095] As an example, the garbage collection GC metadata structure of each ROW object records the cold / hot tag value of that ROW object; When obtaining module 401 performs the process of obtaining the cold / hot flag value of the ROW object that was redirected during a write in at least one bin chain matched by the garbage collection event, it is further configured to: For any ROW object in the bin chain, obtain the cold / hot tag value of the ROW object from the GC metadata structure of the ROW object.

[0096] As an example, the hot / cold tag value for each ROW object is determined based on the last modification timestamp of all data slices that make up the ROW object.

[0097] As one embodiment, the apparatus further includes: an aggregation module; the aggregation module is used for: A new ROW object is generated based on the remaining data slices in the existing ROW object and the data slices to be added to the existing ROW object; and the hot and cold flag values ​​of the new ROW object are determined based on the last modification timestamps of the existing data slices in the new ROW object.

[0098] As one embodiment, when determining the garbage collection efficiency of a ROW object based on its hot / cold labeling value and the proportion of garbage data in that ROW object, the determining module 402 is further configured to: Obtain the percentage of valid data in the ROW object; the sum of the percentage of valid data and the percentage of garbage data in the ROW object is 1. Based on the hot / cold flag value of the ROW object, the creation time of the storage pool, and the current time, the normalized value of the ROW object's hot / cold index is determined; the normalized value of the hot / cold index is negatively correlated with the hot / cold flag value. The waste recycling efficiency of a ROW object is calculated based on the proportion of waste data, the proportion of valid data, and the normalized value of the hot / cold index. Among these, the proportion of waste data is positively correlated with the waste recycling efficiency, the proportion of valid data is negatively correlated with the waste recycling efficiency, and the normalized value of the hot / cold index is positively correlated with the waste recycling efficiency.

[0099] As an example, when performing garbage collection on ROW objects based on their garbage collection efficiency, the recycling module 403 further configures itself to: After obtaining the N bin chains that match the garbage collection event, select one ROW object from each bin chain in a preset order to obtain N ROW objects; N is a positive integer; Select the ROW object with the highest garbage collection efficiency from N ROW objects for garbage collection.

[0100] As an example, obtaining the cold / hot flag value of the write-on-redirect ROW object in at least one bin chain matched by the garbage collection event is performed when the storage pool utilization is less than or equal to a set utilization threshold; When the storage pool utilization rate exceeds the utilization rate threshold, the recycling module 403 is further used for: According to the corresponding recycling level from high to low, traverse at least one bin chain matched by the garbage collection event in turn; the higher the recycling level of any bin chain, the greater the proportion of garbage data of ROW objects on that bin chain. For the currently traversed bin chain, reclaim the ROW objects in the bin chain in the set order.

[0101] This concludes the process. Figure 4 Structural description of the device shown.

[0102] See Figure 5 , Figure 5 This is a structural diagram of an electronic device provided in an embodiment of this application. Figure 5 As shown, the hardware structure may include: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method disclosed in the above example of this application.

[0103] Based on the same application concept as the above method, this application embodiment also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the method disclosed in the above examples of this application.

[0104] For example, the aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For instance, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0105] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A garbage collection method for a distributed storage system, characterized in that, The method includes: When a garbage collection event is detected, the hot / cold flag value of the write-redirected ROW object in at least one bin chain matched by the garbage collection event is obtained; the hot / cold flag value is used to indicate the hot / cold degree of the ROW object; wherein, the smaller the hot / cold flag value, the colder the ROW object, and the larger the hot / cold flag value, the hotter the ROW object. The garbage collection efficiency of a ROW object is determined based on its hot / cold label value and the proportion of garbage data in that ROW object; the smaller the hot / cold label value and the larger the proportion of garbage data, the greater the garbage collection efficiency of the ROW object. ROW objects are recycled based on their garbage collection efficiency.

2. The method according to claim 1, characterized in that, The garbage collection metadata structure of each ROW object records the cold and hot tag values ​​of that ROW object; Obtaining the hot / cold flag values ​​of ROW objects in at least one bin chain matched by the garbage collection event includes: For any ROW object in the bin chain, obtain the hot / cold tag value of the ROW object from the GC metadata structure of the ROW object.

3. The method according to claim 1 or 2, characterized in that, The hot / cold flag value for each ROW object is determined based on the last modification timestamp of all data slices that make up that ROW object.

4. The method according to claim 1 or 2, characterized in that, After a partial data slice in any ROW object has been reclaimed, the method further includes: A new ROW object is generated based on the remaining data slices in the ROW object and the data slices to be added to the ROW object; and the hot / cold flag value of the new ROW object is determined based on the last modification timestamp of the existing data slices in the new ROW object.

5. The method according to claim 1 or 2, characterized in that, The determination of the waste recycling efficiency of a ROW object based on its hot / cold label value and the proportion of waste data in that ROW object includes: Obtain the percentage of valid data in the ROW object; the sum of the percentage of valid data and the percentage of garbage data in the ROW object is 1. Based on the hot / cold flag value of the ROW object, the creation time of the storage pool, and the current time, a normalized value for the hot / cold index of the ROW object is determined; the normalized value for the hot / cold index is negatively correlated with the hot / cold flag value. Based on the proportion of waste data, the proportion of valid data, and the normalized value of the hot / cold index of the ROW object, the waste recycling benefit of the ROW object is calculated; wherein, the proportion of waste data is positively correlated with the waste recycling benefit, the proportion of valid data is negatively correlated with the waste recycling benefit, and the normalized value of the hot / cold index is positively correlated with the waste recycling benefit.

6. The method according to claim 1 or 2, characterized in that, The method of recycling ROW objects based on their garbage collection efficiency includes: After obtaining the N bin chains that match the garbage collection event, select one ROW object from each bin chain in a preset order to obtain N ROW objects, where N is a positive integer; Select the ROW object with the highest garbage collection efficiency from the N ROW objects for recycling.

7. The method according to claim 1 or 2, characterized in that, The process of obtaining the cold / hot flag value of the write-time redirected ROW object in at least one bin chain matched by the garbage collection event is performed when the utilization rate of the storage pool is less than or equal to a set utilization rate threshold. When the utilization rate of the storage pool exceeds the utilization rate threshold, the method further includes: According to the corresponding recycling level from high to low, traverse at least one bin chain that matches the garbage collection event in turn; The higher the recycling level of any bin chain, the greater the proportion of garbage data in the ROW objects on that bin chain. For the currently traversed bin chain, reclaim the ROW objects in the bin chain in the set order.

8. A waste collection device for a distributed storage system, characterized in that, The device includes: The module is used to obtain the hot / cold flag value of the write-redirected ROW object in at least one bin chain matched by the garbage collection event when a garbage collection event is detected; the hot / cold flag value is used to indicate the hot / cold degree of the ROW object; wherein, the smaller the hot / cold flag value, the colder the ROW object, and the larger the hot / cold flag value, the hotter the ROW object. The determination module is used to determine the garbage collection efficiency of a ROW object based on its hot / cold label value and the proportion of garbage data in that ROW object; wherein, the smaller the hot / cold label value and the larger the proportion of garbage data, the greater the garbage collection efficiency of the ROW object. The recycling module is used to recycle ROW objects based on their garbage collection efficiency.

9. The apparatus according to claim 8, characterized in that, The garbage collection metadata structure of each ROW object records the cold and hot tag values ​​of that ROW object; When the obtaining module performs the write-redirection of the ROW object's cold / hot flag value in at least one bin chain matched by the garbage collection event, it is further configured to: For any ROW object in the bin chain, obtain the cold / hot tag value of the ROW object from the GC metadata structure of the ROW object; And / or, The hot / cold flag value for each ROW object is determined based on the last modification timestamp of all data slices that make up that ROW object; And / or, The device further includes: an aggregation module; the aggregation module is used for: After some data slices in any ROW object are recycled, a new ROW object is generated based on the remaining data slices in the ROW object and the data slices to be added to the ROW object; and the hot and cold tag value of the new ROW object is determined according to the last modification timestamp of the existing data slices in the new ROW object. And / or, When determining the garbage collection efficiency of a ROW object based on its hot / cold label value and the proportion of garbage data, the determining module is further configured to: Obtain the percentage of valid data in the ROW object; the sum of the percentage of valid data and the percentage of garbage data in the ROW object is 1. Based on the hot / cold flag value of the ROW object, the creation time of the storage pool, and the current time, a normalized value for the hot / cold index of the ROW object is determined; the normalized value for the hot / cold index is negatively correlated with the hot / cold flag value. Based on the proportion of garbage data, the proportion of valid data, and the normalized value of the hot / cold index of the ROW object, the garbage recycling benefit of the ROW object is calculated; wherein, the proportion of garbage data is positively correlated with the garbage recycling benefit, the proportion of valid data is negatively correlated with the garbage recycling benefit, and the normalized value of the hot / cold index is positively correlated with the garbage recycling benefit; And / or, When performing the garbage collection of ROW objects based on their garbage collection efficiency, the recycling module is further configured to: After obtaining the N bin chains that match the garbage collection event, a ROW object is selected from each bin chain in a preset order to obtain N ROW objects; where N is a positive integer. Select the ROW object with the highest garbage collection efficiency from the N ROW objects for garbage collection; And / or, The process of obtaining the cold / hot flag value of the write-time redirected ROW object in at least one bin chain matched by the garbage collection event is performed when the utilization rate of the storage pool is less than or equal to a set utilization rate threshold. When the utilization rate of the storage pool exceeds the utilization rate threshold, the recycling module is further configured to: According to the corresponding recycling level from high to low, traverse at least one bin chain matched by the garbage collection event in turn; the higher the recycling level of any bin chain, the greater the proportion of garbage data of ROW objects on that bin chain. For the currently traversed bin chain, reclaim the ROW objects in the bin chain in the set order.

10. An electronic device, characterized in that, The electronic device includes: Processor; and A machine-readable storage medium storing machine-executable instructions that, when executed by the processor, cause the processor to perform the steps of the method as described in any one of claims 1 to 7.

11. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions that, when executed by a processor, cause the processor to perform the steps of the method as described in any one of claims 1 to 7.

Images