Hard Disk Garbage Collection Method, Device, Electronic Device, and Storage Medium

Abstract

The present disclosure relates to a hard disk garbage collection method, device, electronic device and storage medium, and relates to the field of computer technology. The method includes: obtaining the data writing amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle; determining the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle and the release bandwidth of the previous cycle based on the redundancy number of each object, the data writing amount and garbage release amount of the current cycle and the previous cycle; dynamically adjusting the garbage collection parameters based on the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle and the release bandwidth of the previous cycle; and performing garbage collection on each storage unit of the hard disk based on the garbage collection parameters. The application of the present disclosure can cope with the fluctuations and changes in the foreground read and write bandwidth, and improve the robustness of the system.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a hard disk garbage recovery method, device, electronic device, and storage medium. Background Art

[0002] To maximize the write performance of hard drives, a common industry practice is to redirect (ROW) data written to disk pool objects, aggregate them, and then write them sequentially to objects allocated by the disk pool. In the distributed storage field, appending to data can significantly improve data write performance. ROW also uses this technique, but appending to the same block of data generates corresponding garbage data. Therefore, the industry typically uses garbage collection algorithms to reclaim garbage space. Since ROW generally uses large objects (such as 128MB per object), a common practice is to scan these objects and perform garbage collection on those whose garbage content meets a certain threshold to free up space.

[0003] However, this solution generally uses fixed garbage collection parameters and cannot perceive changes in foreground writes, resulting in suboptimal space release and low efficiency. Summary of the Invention

[0004] Embodiments of the present disclosure provide a hard disk garbage collection method, apparatus, electronic device, and storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a hard disk garbage collection method, comprising: obtaining the data writing amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle; determining the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle based on the redundant amount of each object, the data writing amount and garbage release amount of the current cycle and the previous cycle; dynamically adjusting the garbage collection parameters based on the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle; and performing garbage collection on each storage unit of the hard disk according to the garbage collection parameters.

[0006] In a second aspect, an embodiment of the present disclosure provides a hard disk garbage collection device, comprising: an acquisition unit, configured to acquire the data writing amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle; a determination unit, configured to determine the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle based on the redundant number of each object, the data writing amount and garbage release amount of the current cycle and the previous cycle; an adjustment unit, configured to dynamically adjust the garbage collection parameters based on the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle; and a recovery unit, configured to perform garbage collection on each storage unit of the hard disk according to the garbage collection parameters.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the hard disk garbage collection method as described in the first aspect is implemented.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hard disk garbage collection method as described in the first aspect.

[0009] By applying the technical solution disclosed in the present invention, the corresponding write bandwidth and release bandwidth can be determined based on the data writing amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle, and the garbage collection parameters can be dynamically adjusted based on the corresponding write bandwidth and release bandwidth. Furthermore, garbage collection can be performed on the hard disk based on the garbage collection parameters, thereby being able to cope with fluctuations and changes in the foreground read and write bandwidth and improving the robustness of the system.

[0010] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0012] Figure 1 A flowchart of an embodiment of the hard disk garbage collection method disclosed herein;

[0013] Figure 2 This is a flow chart of another embodiment of the hard disk garbage collection method disclosed herein;

[0014] Figure 3This is a structural diagram of an embodiment of a hard disk garbage collection device disclosed herein;

[0015] Figure 4 This is a schematic structural diagram of an embodiment of an electronic device disclosed herein. DETAILED DESCRIPTION

[0016] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0017] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0018] In the absence of conflict, the embodiments of the present disclosure and the features thereof may be combined with each other.

[0019] In order to make the technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Figure 1 FIG. 1 shows a process 100 of an embodiment of the hard disk garbage collection method disclosed herein. Figure 1 As shown, the hard disk garbage collection method of this embodiment may include the following steps:

[0021] Step 101: Obtain the data writing amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle.

[0022] In this embodiment, the amount of data written and the amount of garbage released for each object in each storage unit of the hard disk in the current cycle and the previous cycle can be first obtained. Here, the storage unit can be a basic unit of the hard disk, such as a rank, a bank, etc. Objects can be stored in each storage unit. An object can include multiple copies. The amount of data written and the amount of garbage released for each object in the current cycle and the previous cycle can be obtained through multiple preset functions. The user can pre-set the duration of a cycle and the start time of the cycle. The amount of data written is the amount of data written to the storage unit of the hard disk, and the amount of garbage released is the amount of invalid data identified.

[0023] Step 102 : Determine the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle based on the redundancy number of each object, the data write amount and the garbage release amount of the current cycle and the previous cycle.

[0024] After obtaining the data write amount and garbage release amount in the current cycle, as well as the data write amount and garbage release amount in the previous cycle, the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle can be further calculated. Specifically, the calculation can be based on the redundancy of each object. For example, the data write amount can be divided by the redundancy to obtain the write bandwidth. The garbage release amount can be divided by the redundancy to obtain the release bandwidth. Alternatively, the write bandwidth can be further determined based on the redundancy type of the object.

[0025] Step 103 : Dynamically adjust garbage collection parameters according to the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle.

[0026] After calculating the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle, the garbage collection parameters can be dynamically adjusted according to the above four parameters. Specifically, the write bandwidth of the current cycle can be compared with the write bandwidth of the previous cycle, and the release bandwidth of the current cycle can be compared with the release bandwidth of the previous cycle. Adjust the garbage collection parameters based on the comparison results of the two. Alternatively, compare the write bandwidth of the current cycle with the release bandwidth of the current cycle, and adjust the garbage collection parameters based on the comparison results. Garbage collection parameters may include but are not limited to: the concurrent number of garbage collection objects, the concurrent number of reverse query data, and the concurrent number of reads, etc.

[0027] In some specific practices, when the write bandwidth of the current cycle is greater than the write bandwidth of the previous cycle, and the release bandwidth of the current cycle is greater than the release bandwidth of the previous cycle, it is considered that excessive garbage release will affect the write performance. In this case, the garbage collection parameters can be reduced to schedule more resources to the data writing task, thereby reducing the impact on the write performance.

[0028] Step 104: performing garbage collection on each storage unit of the hard disk according to the garbage collection parameters.

[0029] After adjusting the garbage collection parameters, garbage collection can be performed on each storage unit of the hard drive based on the garbage collection parameters. Specifically, invalid data in each storage unit of the hard drive is identified based on the garbage collection parameters. When the amount of invalid data reaches a preset threshold, the valid data in a single storage unit on the hard drive is moved to other storage units, and the invalid data is deleted, thus achieving garbage collection on the hard drive.

[0030] The hard disk garbage collection method provided by the above-mentioned embodiments of the present disclosure can determine the corresponding write bandwidth and release bandwidth according to the data writing amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle, and dynamically adjust the garbage collection parameters according to the corresponding write bandwidth and release bandwidth, and further perform garbage collection on the hard disk according to the garbage collection parameters, so as to cope with fluctuations and changes in the foreground read and write bandwidth, thereby improving the robustness of the system.

[0031] Continue to see Figure 2 , which shows a process 200 of another embodiment of the hard disk garbage collection method according to the present disclosure. Figure 2 As shown, the method in this embodiment may include the following steps:

[0032] Step 201: Obtain the data writing amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle.

[0033] Step 202: Determine the redundancy type and redundancy quantity of each object; determine the write bandwidth of the current cycle based on the redundancy type, redundancy quantity of each object and the data write amount of the current cycle; determine the release bandwidth of the current cycle based on the redundancy type, redundancy quantity of each object and the garbage release amount of the current cycle; determine the write bandwidth of the previous cycle based on the redundancy type, redundancy quantity of each object and the data write amount of the previous cycle; determine the release bandwidth of the previous cycle based on the redundancy type, redundancy quantity of each object and the garbage release amount of the previous cycle.

[0034] In this embodiment, the redundancy type and amount of each object can be determined first. Redundancy types can include replicas and erasure code redundancy. Replication refers to the redundant storage of data, where multiple independent copies of the data are stored on different servers. This approach ensures data reliability and availability. Even if a server fails, other copies of the data can still be accessed and used. Erasure code redundancy provides fault tolerance by dividing the original data into multiple data blocks and encoding these blocks to generate redundant data blocks (check blocks).

[0035] After determining the redundancy type and amount, the write bandwidth and release bandwidth can be determined based on the amount of data written and the amount of garbage released. Specifically, the write bandwidth of the current cycle can be determined based on the redundancy type and amount of redundancy of each object and the amount of data written in the current cycle. If the redundancy type of each object is replica, the number of replicas is 3. If the amount of data written to the hard disk by the object is 3T, the actual write bandwidth is 3T / 3=1T. If the redundancy type of each object is erasure code redundancy, the number of redundancies is 4 blocks of original data elements and 2 blocks of redundant elements. If the amount of data written to the hard disk by the object is 3T, the actual write bandwidth is 3T*4 / (4+2)=2T.

[0036] Similarly, the release bandwidth of the current cycle can be determined based on the redundancy type, redundancy quantity and garbage release amount of each object in the current cycle; the write bandwidth of the previous cycle can be determined based on the redundancy type, redundancy quantity and data write amount of each object in the previous cycle; and the release bandwidth of the previous cycle can be determined based on the redundancy type, redundancy quantity and garbage release amount of each object in the previous cycle.

[0037] Step 203: determine a first difference between the write bandwidth of the current cycle and the write bandwidth of the previous cycle; determine a second difference between the release bandwidth of the current cycle and the release bandwidth of the previous cycle; and dynamically adjust garbage collection parameters based on the first difference and the second difference.

[0038] After calculating the write bandwidth of the current cycle, the write bandwidth of the previous cycle, the release bandwidth of the current cycle, and the release bandwidth of the previous cycle, a first difference between the write bandwidth of the current cycle and the write bandwidth of the previous cycle can be calculated, as well as a second difference between the release bandwidth of the current cycle and the release bandwidth of the previous cycle. Garbage collection parameters can then be dynamically adjusted based on the first and second differences.

[0039] Specifically, if the first difference and the second difference are close to 0, it means that the write bandwidth of the current cycle is equal to the write bandwidth of the previous cycle, the release bandwidth of the current cycle is equal to the release bandwidth of the previous cycle, and the current state is in equilibrium, so there is no need to adjust the garbage collection parameters. If the first difference is less than the first preset threshold and the second difference is greater than the second preset threshold, it means that the current garbage collection state is occupying too many resources, and the garbage collection parameters need to be lowered to reduce the resources occupied by garbage collection.

[0040] In some optional implementations of this embodiment, if the first difference is less than a first preset threshold and the second difference is greater than a second preset threshold, the garbage collection parameter may be lowered. If the error between the first difference and the preset value is less than a third preset threshold and the second difference is less than a fourth preset threshold, the garbage collection parameter may be increased.

[0041] In this implementation, if the first difference is less than a first preset threshold and the second difference is greater than a second preset threshold, it indicates that the current garbage collection is occupying too many resources and the garbage collection parameters need to be lowered to reduce the resources occupied by garbage collection. If the first difference is equal to a preset value (here, the preset value can be 0) and the second difference is less than a fourth preset threshold, it indicates that the current garbage collection is occupying fewer resources and the garbage collection parameters can be increased.

[0042] In some optional implementations of this embodiment, if the relative error between the write bandwidth of the current cycle and the release bandwidth of the current cycle is less than a fifth preset threshold (the fifth preset threshold may be 0), that is, the amount of data written in the current cycle is equivalent to the amount of garbage released, then the garbage collection parameters may be kept unchanged.

[0043] Step 204, based on the garbage collection parameters, reverse check the metadata of each object to determine whether each object is valid data; in response to determining that each object is valid data, move the object to another location and modify the metadata of the object; in response to determining that each object is invalid data, release the storage space occupied by the object.

[0044] After adjusting the garbage collection parameters, the metadata of each object can be checked against the garbage collection unit number to determine whether the object is valid data. If the object is determined to be valid data, it can be moved to another location. At the same time, the object's metadata is modified. Specifically, the object's new storage address is stored in the updated metadata. If the object is invalid data, the storage space occupied by the object can be released.

[0045] In some optional implementations of this embodiment, the user-set cycle duration and cycle start time can also be obtained. Based on the cycle duration and cycle start time, the start and end times of data acquisition for each cycle are determined. For example, if the user sets a cycle duration of 30 minutes and a start time of 00:00, data acquisition can begin at 00:00 and end at 00:30. The next cycle will start at 00:31 and end at 01:00.

[0046] The hard disk garbage collection method provided by the above-mentioned embodiment of the present disclosure can automatically calculate and compare the foreground write bandwidth and the background release bandwidth, adaptively adjust the garbage collection parameters to achieve a balance between the two, and can cope with fluctuations and changes in the foreground write bandwidth, thereby ensuring the continuous stability of the cluster capacity, improving the robustness of the system, and reducing human maintenance and interference costs.

[0047] Further references Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a hard disk garbage collection device. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0048] like Figure 3 As shown, the hard disk garbage collection device 300 of this embodiment includes: an acquisition unit 301 , a determination unit 302 , an adjustment unit 303 and a collection unit 304 .

[0049] The acquisition unit 301 is configured to acquire the data writing amount and garbage releasing amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle.

[0050] The determination unit 302 is configured to determine the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle according to the redundancy number of each object, the data writing amount and the garbage release amount of the current cycle and the previous cycle.

[0051] The adjustment unit 303 is configured to dynamically adjust the garbage collection parameters according to the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle.

[0052] The recycling unit 304 is configured to perform garbage collection on each storage unit of the hard disk according to the garbage collection parameters.

[0053] In addition, in the technical solution of this application, an electronic device is also proposed.

[0054] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0055] like Figure 4 As shown, the electronic device may include a processor 401, a memory 402, a bus 403, and a computer program stored in the memory 402 and executable on the processor 401, wherein the processor 401 and the memory 402 communicate with each other via the bus 403. When the processor 401 executes the computer program, the steps of the above method are implemented, for example, including: obtaining the data write amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle; determining the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle based on the redundancy amount of each object, the data write amount, and the garbage release amount of the current cycle and the previous cycle; dynamically adjusting the garbage collection parameters based on the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle; and performing garbage collection on each storage unit of the hard disk based on the garbage collection parameters.

[0056] In addition, in one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented, for example, including: obtaining the data writing amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle; determining the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle based on the redundant number of each object, the data writing amount and the garbage release amount of the current cycle and the previous cycle; dynamically adjusting the garbage collection parameters based on the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle; and performing garbage collection on each storage unit of the hard disk according to the garbage collection parameters.

[0057] To sum up, in the technical solution disclosed herein, the corresponding write bandwidth and release bandwidth can be determined based on the data writing amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle, and the garbage collection parameters can be dynamically adjusted according to the corresponding write bandwidth and release bandwidth. Further, garbage collection is performed on the hard disk according to the garbage collection parameters, so as to cope with fluctuations and changes in the foreground read and write bandwidth and improve the robustness of the system.

[0058] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A hard disk garbage collection method, comprising: Get the amount of data written and garbage released for each object in each storage unit of the hard disk in the current cycle and the previous cycle; Determine the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle based on the redundant amount of each object, the data written amount, and the garbage released amount in the current cycle and the previous cycle, wherein the write bandwidth is the ratio of the data written amount to the redundant amount, and the release bandwidth is the ratio of the garbage released amount to the redundant amount; Dynamically adjusting garbage collection parameters according to the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle; Garbage collection is performed on each storage unit of the hard disk according to the garbage collection parameters.

2. The method according to claim 1, wherein Based on the redundancy of each object, the amount of data written and garbage released in the current and previous cycles, the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle are determined, including: Determine the redundancy type and amount of each object; Determine the write bandwidth for the current cycle based on the redundancy type and redundancy quantity of each object and the data write volume for the current cycle; Determine the release bandwidth of the current cycle based on the redundancy type and amount of each object and the garbage release amount of the current cycle; Determine the write bandwidth for the previous cycle based on the redundancy type and redundancy quantity of each object and the data written in the previous cycle; The release bandwidth of the previous cycle is determined according to the redundancy type and redundancy quantity of each object and the garbage release amount of the previous cycle.

3. The method according to claim 1, wherein The dynamically adjusting the garbage collection parameters according to the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle includes: Determine a first difference between a write bandwidth of a current cycle and a write bandwidth of a previous cycle; Determine a second difference between the released bandwidth of the current cycle and the released bandwidth of the previous cycle; The garbage collection parameter is dynamically adjusted according to the first difference and the second difference.

4. The method according to claim 3, wherein: The dynamically adjusting the garbage collection parameter according to the first difference and the second difference includes: In response to determining that the first difference is less than a first preset threshold and the second difference is greater than a second preset threshold, reducing the garbage collection parameter; In response to determining that the error between the first difference and the preset value is less than a third preset threshold and the second difference is less than a fourth preset threshold, the garbage collection parameter is increased.

5. The method according to claim 4, wherein The dynamically adjusting the garbage collection parameter according to the first difference and the second difference includes: In response to determining that a relative error between the write bandwidth of the current cycle and the release bandwidth of the current cycle is less than a fifth preset threshold, the garbage collection parameter is kept unchanged.

6. The method according to claim 1, wherein The performing garbage collection on each storage unit of the hard disk according to the garbage collection parameter includes: According to the garbage collection parameters, the metadata of each object is checked to determine whether each object is valid data; In response to determining that each object is valid data, moving the object to another location and modifying metadata of the object; In response to determining that each object is invalid data, the storage space occupied by the object is released.

7. The method according to claim 1, wherein The method further comprises: Get the cycle duration and cycle start time set by the user; The start time and end time of data acquisition of each cycle are determined according to the cycle duration and the cycle start time.

8. A hard disk garbage collection device, comprising: An acquisition unit is configured to acquire the amount of data written and the amount of garbage released for each object in each storage unit of the hard disk in the current cycle and the previous cycle; a determining unit configured to determine a write bandwidth of a current cycle, a release bandwidth of a current cycle, a write bandwidth of a previous cycle, and a release bandwidth of a previous cycle based on the redundancy number of each object, the data write amount, and the garbage release amount in the current cycle and the previous cycle, wherein the write bandwidth is a ratio of the data write amount to the redundancy number, and the release bandwidth is a ratio of the garbage release amount to the redundancy number; an adjusting unit configured to dynamically adjust garbage collection parameters according to the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle; The recycling unit is configured to perform garbage collection on each storage unit of the hard disk according to the garbage collection parameter.

9. An electronic device comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the hard disk garbage collection method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the hard disk garbage collection method according to any one of claims 1 to 7 is implemented.

Images