Method for recycling and storing garbage and storage equipment
By monitoring the effective data capacity of stored objects and dynamically adjusting the garbage collection strategy, the risk caused by high disk pool capacity in write-intensive storage systems is resolved, thereby improving the stability and performance of the storage system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NEW H3C TECH CO LTD
- Filing Date
- 2025-12-26
- Publication Date
- 2026-04-28
AI Technical Summary
In write-intensive storage systems, existing garbage collection mechanisms reduce the storage system's ability to withstand risks when the disk pool capacity reaches a high level, affecting the IO performance of front-end business applications and consuming a lot of resources.
By monitoring the effective data capacity of stored objects, the target reclamation tier, reclamation quota, and number of ULTs are adjusted to dynamically adjust the garbage collection strategy, thereby reducing the disk pool capacity and avoiding high water levels.
It effectively reduces the risk of disk pool capacity issues, improves the stability of the storage system and the IO performance of front-end business operations, and reduces resource consumption.
Smart Images
Figure CN121934767A_ABST
Abstract
Description
Technical Field
[0001] This specification relates to the field of communication technology, and in particular to a method and storage device for recycling and storing waste. Background Technology
[0002] GC: Garbage Collection.
[0003] ROW (Redirect on Write): When writing data, a new address space is allocated to overwrite the data, and the data index is redirected to the new offset. The old data remains in its original location.
[0004] ULT: Use-level Thread.
[0005] IO: IN / OUT, in the storage domain, refers to reading and writing stored data.
[0006] In the storage field, especially in write-intensive storage systems, a Row-of-Work (ROW) mechanism is often introduced to improve data write performance, particularly small I / O writes. However, in write-intensive storage systems, ROW generates a large number of old data blocks that cannot be referenced, resulting in occupied storage space. Therefore, ROW needs to be accompanied by a garbage collection (GC) mechanism. GC determines which data blocks can be safely reclaimed by scanning metadata dependencies. During garbage collection, certain resources (such as CPU and bandwidth) are consumed, impacting the performance of foreground business I / O. GC's flow control mechanism aims to reclaim resources as much as possible while ensuring that business operations are not excessively affected. Summary of the Invention
[0007] To overcome the problems existing in related technologies, this specification provides a method and storage device for recycling and storing waste.
[0008] According to a first aspect of the embodiments of this specification, a method for recycling stored waste is provided, the method comprising: Get the effective data capacity of the storage object; Based on the available data capacity, adjust the target recycling tier, target recycling quota, and target ULT quantity for the corresponding object. Waste is collected based on the target recycling level, target recycling quota, and target ULT quantity.
[0009] The step of obtaining the effective data capacity of the storage object includes: The effective data capacity of an object is determined based on the amount of garbage in its corresponding metadata.
[0010] The step of adjusting the target recycling level corresponding to the object based on the effective data capacity includes: Calculate the percentage of effective data capacity based on the effective data capacity. The first target level is determined based on the proportion of effective data volume. Adjust the target recycling level corresponding to the first target.
[0011] The adjustment of the target recovery quota based on the effective data capacity includes: Calculate the percentage of effective data capacity based on the effective data capacity. The second target level is determined based on the proportion of effective data volume. Adjust the target recovery quota corresponding to the target object according to the second target.
[0012] The target number of ULTs is adjusted based on the effective data capacity, including: Calculate the percentage of effective data capacity based on the effective data capacity. The third target tier is determined based on the proportion of effective data volume. The target ULT quantity corresponding to the third target is adjusted. As can be seen from the above embodiments, by judging the proportion of effective data capacity and using different garbage collection control levels, the disk pool capacity is reduced during garbage collection speed adjustment, so that the disk pool capacity is not easily kept at a high level, reducing the risk of high storage capacity usage.
[0013] According to a second aspect of the embodiments of this specification, a storage device is provided, the storage device comprising: The acquisition module is used to obtain the effective data capacity of the stored object; The processing module is used to adjust the target recycling level, target recycling quota, and target ULT quantity corresponding to the object based on the effective data capacity. The processing module is also used to perform waste recycling on objects based on the target recycling level, target recycling quota, and target ULT quantity.
[0014] Specifically, the acquisition module is used to determine the effective data capacity of an object based on the amount of garbage in the object's corresponding metadata.
[0015] Specifically, the processing module is used to calculate the percentage of effective data capacity based on the effective data capacity. The first target level is determined based on the proportion of effective data volume. Adjust the target recycling level corresponding to the first target.
[0016] Specifically, the processing module is used to calculate the percentage of effective data capacity based on the effective data capacity. The second target level is determined based on the proportion of effective data volume. Adjust the target recovery quota corresponding to the target object according to the second target.
[0017] Specifically, the processing module is used to calculate the percentage of effective data capacity based on the effective data capacity. The third target tier is determined based on the proportion of effective data volume. Adjust the number of target ULTs corresponding to the third target object.
[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description
[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.
[0020] Figure 1 This is a flowchart illustrating a method for recycling and storing waste according to an exemplary embodiment. Detailed Implementation
[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.
[0022] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.
[0023] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."
[0024] In the existing garbage collection mechanism, in order to reduce the impact on normal business operations, the GC collection intensity is controlled through the minimum collection level, collection quota, and ULT quantity.
[0025] Minimum recycling level: At the data object level (generally, after a file is written to a distributed cluster, it will be divided into objects of fixed size according to a certain granularity), metadata on the amount of garbage is set for each object, ranging from 1M to 10 8K, and divided into bin0 to bin50. Objects in bin0 do not contain garbage, while objects in bin50 are all garbage, with each level divided into 2% granularity.
[0026] As shown in Table 1, different hard disk utilization rates correspond to different minimum recycling levels for the control recycling strategy and the minimum recycling level for the idle time strategy. For example, when the hard disk utilization rate is in [0, 20), the corresponding minimum recycling level for the control recycling strategy is bin3 (calculated at a granularity of 2%, that is, garbage collection is only performed when the amount of garbage reaches more than 6% during non-idle periods; and garbage collection is only performed when the amount of garbage reaches more than 98% during idle periods).
[0027] Table 1 Recycling quota: (details) See the table below, as shown in Table 2.
[0028] The rate control and recycling strategy calculates the recycling quota based on the foreground write bandwidth: Off-peak recycling strategy, calculating recycling quotas based on engine waste volume: Table 2 The number of ULTs is shown in Table 3. When the disk pool utilization is below 90%, the maximum number of ULTs reclaimed per GC instance is 8, ult_max=8. When the disk pool utilization is above 90%, ult_max is dynamically adjusted based on the difference between the front-end write bandwidth write_bw and the GC recycling bandwidth recycle_bw, as well as the engine garbage size engine_garbage_size. default It is 10M and the magnification is 5.
[0029] Idle time recycling strategy: When the disk pool utilization rate is below 90%, the maximum number of ults that can be reclaimed by each GC instance is 2, ult_max=2. When the disk pool utilization rate is above 90%, the maximum number of ults that can be reclaimed by each GC instance is 8, ult_max=8.
[0030] Table 3 However, the existing approach has the following problems: during data writing, the amount of garbage in the storage cluster increases until the disk pool capacity reaches 90%, at which point write-and-reclaim balance is achieved. When the capacity level of the distributed storage cluster is high, the storage system's ability to withstand risks decreases significantly; for example, data reconstruction may not be possible after a single point of failure in the distributed storage cluster.
[0031] To address the aforementioned technical problems, embodiments of this disclosure provide a method for recycling and storing waste, such as... Figure 1 As shown, the method includes: S101 Obtains the effective data capacity of the storage object; S102 adjusts the target recycling level, target recycling quota, and target ULT quantity corresponding to the object based on the effective data capacity; S103 performs waste recycling on objects based on the target recycling level, target recycling quota, and target ULT quantity.
[0032] In this embodiment, an object is a data storage model that packages data, metadata, and unique identifiers into a single, self-contained unit for management.
[0033] An object typically contains three core parts: data, metadata, and a globally unique identifier (ID).
[0034] In step S101, the effective data capacity can be obtained by monitoring the objects. For example, if the monitoring shows that the BIN of object 1 is 0, it can be known that there is no garbage in object 1, that is, its effective data capacity accounts for 100%. Here, BIN represents the metadata of the amount of garbage set for each object at the data object level, which is divided into bin0~bin50. Objects in bin0 do not contain garbage, and objects in bin50 are all garbage. Each level is divided into 2% granularity.
[0035] The effective data capacity of each object can be obtained using the methods described above.
[0036] In step S102, different preset levels can be set for recycling level, recycling quota, and ULT quantity.
[0037] For example, in determining the target recycling level, three levels are set, as shown in Table 4. For different hard disk utilization rates, three levels are set for the effective data capacity percentage, namely [0,50), [50,90), and [90,100]. Different effective data capacity percentages can correspond to different minimum recycling levels for control recycling strategies and minimum recycling levels for idle time strategies. Assuming that the effective data capacity percentage of object 1 is 60 (i.e., the first target level) and the hard disk utilization rate is [0,20), then the minimum recycling level for control recycling strategies is BIN2 (i.e., the target recycling level), and the minimum recycling level for idle time strategies is bin48 (i.e., the target recycling level).
[0038] Table 4 For example, in determining the target recycling quota, three levels are set, as shown in Table 5. For different hard drive utilization rates, three levels are set for the effective data capacity ratio, namely [0,50), [50,90), and [90,100]. Different effective data capacity ratios can correspond to different control recycling strategy quota coefficients and idle time strategy quota coefficients. Assuming that the effective data capacity ratio of object 1 is 60 (i.e., the first target level) and the hard drive utilization rate is in the range of [0,20), then the control recycling strategy quota coefficient is 15 (i.e., the target recycling level), and the lowest recycling level of the idle time strategy is 15 (i.e., the target recycling level).
[0039] Table 5 For example, in determining the ULT number, three levels are set, as shown in Table 6. For different hard disk utilization rates, three levels are set for the effective data capacity ratio, namely [0,50), [50,90), and [90,100]. Different effective data capacity ratios can correspond to different ult_max and Ult_max amplification factors. Assuming that the effective data capacity ratio of object 1 is 60 (i.e., the first target level), and the hard disk utilization rate is in the range of [0,20), then the ult_max of the concurrent control of the speed control recycling strategy is 8, and the Ult_max amplification factor is 2. The ult_max of the idle time strategy quota coefficient is 8, and the Ult_max amplification factor is 1 (i.e., the target recycling level).
[0040] Table 6 It should be noted that the effective data capacity mentioned above can be set according to user needs, and this embodiment is only for illustrative purposes.
[0041] In this embodiment, an active garbage collection mechanism can also be set, and a threshold can be set. When the effective data capacity is less than the threshold (e.g., 50%), the storage system will report an alarm, and the user can actively clean up the garbage.
[0042] As can be seen from the above embodiments, by judging the proportion of effective data capacity and using different garbage collection control levels, the disk pool capacity can be reduced when the garbage collection speed is adjusted, so that the disk pool capacity will not easily be at a high level, thus reducing the risk of high storage capacity usage.
[0043] Based on the above method embodiments, this disclosure also provides a storage device, the storage device comprising: The acquisition module is used to obtain the effective data capacity of the stored object; The processing module is used to adjust the target recycling level, target recycling quota, and target ULT quantity corresponding to the object based on the effective data capacity. The processing module is also used to perform waste recycling on objects based on the target recycling level, target recycling quota, and target ULT quantity.
[0044] Specifically, the acquisition module is used to determine the effective data capacity of an object based on the amount of garbage in the object's corresponding metadata.
[0045] Specifically, the processing module is used to calculate the percentage of effective data capacity based on the effective data capacity. The first target level is determined based on the proportion of effective data volume. Adjust the target recycling level corresponding to the first target.
[0046] Specifically, the processing module is used to calculate the percentage of effective data capacity based on the effective data capacity. The second target level is determined based on the proportion of effective data volume. Adjust the target recovery quota corresponding to the target object according to the second target.
[0047] Specifically, the processing module is used to calculate the percentage of effective data capacity based on the effective data capacity. The third target tier is determined based on the proportion of effective data volume. Adjust the number of target ULTs corresponding to the third target object.
[0048] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0049] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0050] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this specification are indicated by the following claims.
[0051] It should be understood that this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is limited only by the appended claims.
[0052] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.
Claims
1. A method for recycling and storing waste, characterized in that, The method includes: Get the effective data capacity of the storage object; Based on the available data capacity, adjust the target recycling tier, target recycling quota, and target ULT quantity for the corresponding object. Waste is collected based on the target recycling level, target recycling quota, and target ULT quantity.
2. The method according to claim 1, characterized in that, The process of obtaining the effective data capacity of the storage object includes: The effective data capacity of an object is determined based on the amount of garbage in its corresponding metadata.
3. The method according to claim 1, characterized in that, The step of adjusting the target recycling level corresponding to the object based on the effective data capacity includes: Calculate the percentage of effective data capacity based on the effective data capacity. The first target level is determined based on the proportion of effective data volume. Adjust the target recycling level corresponding to the first target.
4. The method according to claim 1, characterized in that, Adjustments to the target recovery quota based on available data capacity include: Calculate the percentage of effective data capacity based on the effective data capacity. The second target level is determined based on the proportion of effective data volume. Adjust the target recovery quota corresponding to the target object according to the second target.
5. The method according to claim 1, characterized in that, Adjust the target number of ULTs based on the effective data capacity, including: Calculate the percentage of effective data capacity based on the effective data capacity. The third target tier is determined based on the proportion of effective data volume. Adjust the number of target ULTs corresponding to the third target object.
6. A storage device, characterized in that, The storage device includes: The acquisition module is used to obtain the effective data capacity of the stored object; The processing module is used to adjust the target recycling level, target recycling quota, and target ULT quantity corresponding to the object based on the effective data capacity. The processing module is also used to perform waste recycling on objects based on the target recycling level, target recycling quota, and target ULT quantity.
7. The storage device according to claim 6, characterized in that, The acquisition module is specifically used to determine the effective data capacity of an object based on the amount of garbage in the object's corresponding metadata.
8. The storage device according to claim 6, characterized in that, The processing module is specifically used to calculate the percentage of effective data capacity based on the effective data capacity. The first target level is determined based on the proportion of effective data volume. Adjust the target recycling level corresponding to the first target.
9. The storage device according to claim 6, characterized in that, The processing module is specifically used to calculate the percentage of effective data capacity based on the effective data capacity. The second target level is determined based on the proportion of effective data volume. Adjust the target recovery quota corresponding to the target object according to the second target.
10. The storage device according to claim 6, characterized in that, The processing module is specifically used to calculate the percentage of effective data capacity based on the effective data capacity. The third target tier is determined based on the proportion of effective data volume. Adjust the number of target ULTs corresponding to the third target object.