Hard disk garbage collection method and device, electronic equipment and storage medium

Abstract

The invention relates to a hard disk garbage collection method and device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring data write-in quantity and garbage release quantity of each object in each storage unit of a hard disk in a current period and a previous period; determining the writing bandwidth of the current period, the releasing bandwidth of the current period, the writing bandwidth of the previous period and the releasing bandwidth of the previous period according to the redundancy quantity of each object and the data writing quantity and the garbage releasing quantity of the current period and the previous period; according to the write-in bandwidth of the current period, the release bandwidth of the current period, the write-in bandwidth of the previous period and the release bandwidth of the previous period, dynamically adjusting garbage collection parameters; and according to the garbage collection parameters, garbage collection is performed on each storage unit of the hard disk. According to the invention, the fluctuation and change of the foreground read-write bandwidth can be coped with, and the robustness of the system is improved.

Description

Technical Field

[0001] This application relates to the field of computer technologies, and in particular, to a hard disk garbage collection method, apparatus, electronic device, and storage medium. Background Art

[0002] To better utilize the write performance of a hard disk, a common practice in the industry is to redirect (ROW) the data written to the hard disk pool objects together, aggregate them, and then sequentially write them to the objects allocated by the hard disk pool. In the field of distributed storage, by adopting the append write method, the write performance of data can be greatly improved. ROW also uses this technology, but when appending the same piece of data, corresponding garbage data will be generated. Therefore, the industry usually adopts a garbage collection algorithm to recycle the garbage space. Since ROW generally uses the technology of large objects (such as an object of 128M), the general approach is to scan these objects and perform garbage collection on the objects that meet the threshold of the garbage amount, so as to achieve the purpose of releasing space.

[0003] However, the parameters for garbage collection in this solution are generally fixed and cannot sense the changes in the foreground write, resulting in unsatisfactory space release and low efficiency. Summary of the Invention

[0004] Embodiments of the present disclosure provide a hard disk garbage collection method, apparatus, electronic device, and storage medium.

[0005] In a first aspect, embodiments of the present disclosure provide a hard disk garbage collection method, including: obtaining the data write amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle; determining the write bandwidth in the current cycle, the release bandwidth in the current cycle, the write bandwidth in the previous cycle, and the release bandwidth in the previous cycle according to the redundancy quantity of each object, the data write amount and garbage release amount in the current cycle and the previous cycle; dynamically adjusting the garbage collection parameters according to the write bandwidth in the current cycle, the release bandwidth in the current cycle, the write bandwidth in the previous cycle, and the release bandwidth in the previous cycle; and performing garbage collection on each storage unit of the hard disk according to the garbage collection parameters.

[0006] In a second aspect, an embodiment of the present disclosure provides a hard disk garbage collection device, including: an acquisition unit configured to acquire the data write amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle; a determination unit configured to determine the write bandwidth in the current cycle, the release bandwidth in the current cycle, the write bandwidth in the previous cycle, and the release bandwidth in the previous cycle according to the redundancy quantity of each object, the data write amount and garbage release amount in the current cycle and the previous cycle; an adjustment unit configured to dynamically adjust the garbage collection parameters according to the write bandwidth in the current cycle, the release bandwidth in the current cycle, the write bandwidth in the previous cycle, and the release bandwidth in the previous cycle; and a recovery unit configured to perform garbage collection on each storage unit of the hard disk according to the garbage collection parameters.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device, including a memory, a processor, a bus, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the hard disk garbage collection method described in the first aspect is implemented.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the hard disk garbage collection method described in the first aspect is implemented.

[0009] By applying the technical solution of the present disclosure, the corresponding write bandwidth and release bandwidth can be determined according to the data write amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle, and the garbage collection parameters can be dynamically adjusted according to the corresponding write bandwidth and release bandwidth. Further, garbage collection is performed on the hard disk according to the garbage collection parameters, so as to be able to cope with the fluctuations and changes of the foreground read / write bandwidth and improve the robustness of the system.

[0010] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them: Figure 1 is a schematic flowchart of an embodiment of the hard disk garbage collection method of the present disclosure; Figure 2 is a schematic flowchart of another embodiment of the hard disk garbage collection method of the present disclosure; Figure 3 is a schematic structural diagram of an embodiment of the hard disk garbage collection device of the present disclosure; Figure 4 Schematic diagram of the structure of an embodiment of the electronic device of the present disclosure. Detailed implementation manners

[0012] It should be noted that the following detailed description is exemplary and is intended to provide further illustration of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present disclosure belongs.

[0013] It should be noted that the terms used herein are only for describing the specific implementation manners and are not intended to limit the exemplary implementation manners according to the present disclosure. As used herein, unless otherwise clearly specified in the context, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0014] In the case of no conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0015] In order to make the technical solutions and advantages of the present disclosure more clear and understandable, the present disclosure will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] Figure 1 Flow 100 of an embodiment of the hard disk garbage collection method of the present disclosure is shown. As Figure 1 shown, the hard disk garbage collection method of this embodiment may include the following steps: Step 101, obtain the data write amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle.

[0017] In this embodiment, the data write amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle may be obtained first. Here, the storage unit may be the basic unit of the hard disk, for example, it may be rank, bank, etc. Each storage unit may store objects. An object may include multiple copies. The data write amount and garbage release amount of each object in the current cycle and the previous cycle may be obtained through a preset plurality of functions. The user may preset the duration of a cycle and the start time of the cycle. The data write amount is the amount of data written into the storage unit of the hard disk, and the garbage release amount is the amount of invalid data identified.

[0018] Step 102, determine the write bandwidth in the current cycle, the release bandwidth in the current cycle, the write bandwidth in the previous cycle, and the release bandwidth in the previous cycle according to the redundancy quantity of each object, the data write amount and garbage release amount in the current cycle and the previous cycle.

[0019] After obtaining the data write volume and garbage release volume of the current cycle, as well as the data write volume and garbage release volume of the previous cycle, the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle can be further calculated. Specifically, when calculating, it can be calculated according to the redundancy quantity of each object. For example, the data write volume can be divided by the redundancy quantity to obtain the write bandwidth. The garbage release volume can be divided by the redundancy quantity to obtain the release bandwidth. Or, the redundancy type of the object can be further combined to determine the write bandwidth.

[0020] Step 103: Dynamically adjust the garbage collection parameters according to the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle.

[0021] After calculating the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle, the garbage collection parameters can be dynamically adjusted according to the above four parameters. Specifically, the write bandwidth of the current cycle can be compared with the write bandwidth of the previous cycle, and at the same time, the release bandwidth of the current cycle can be compared with the release bandwidth of the previous cycle. The garbage collection parameters are adjusted according to the comparison results of the two. Or, the write bandwidth of the current cycle and the release bandwidth of the current cycle can be compared, and the garbage collection parameters are adjusted according to the comparison results. The garbage collection parameters can include but are not limited to: the concurrent quantity of garbage collection objects, the concurrent quantity of reverse lookup data, and the read concurrent quantity, etc.

[0022] In some specific practices, when the write bandwidth of the current cycle is greater than the write bandwidth of the previous cycle and the release bandwidth of the current cycle is greater than the release bandwidth of the previous cycle, it is considered that excessive garbage release in the current cycle will affect the write performance. Then, the garbage collection parameters can be reduced, so that more resources can be scheduled to the data write task, thereby reducing the impact on the write performance.

[0023] Step 104: Perform garbage collection on each storage unit of the hard disk according to the garbage collection parameters.

[0024] After adjusting the garbage collection parameters, garbage collection can be performed on each storage unit of the hard disk according to the garbage collection parameters. Specifically, according to the garbage collection parameters, the invalid data in each storage unit of the hard disk is identified. When the amount of invalid data reaches a preset threshold, the valid data in a single storage unit of the hard disk is moved to other storage units, and the invalid data is deleted, thereby realizing garbage collection of the hard disk.

[0025] The hard disk garbage collection method provided by the above embodiments of the present disclosure can determine the corresponding write bandwidth and release bandwidth according to the data write amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle, and dynamically adjust the garbage collection parameters according to the corresponding write bandwidth and release bandwidth, and further perform garbage collection on the hard disk according to the garbage collection parameters, so as to cope with the fluctuations and changes of the foreground read and write bandwidth, and improve the robustness of the system.

[0026] Continue to refer to Figure 2 , which shows the flow 200 of another embodiment of the hard disk garbage collection method according to the present disclosure. As Figure 2 shown, the method in this embodiment may include the following steps: Step 201, obtain the data write amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle.

[0027] Step 202, determine the redundancy type and redundancy quantity of each object; determine the write bandwidth of the current cycle according to the redundancy type, redundancy quantity of each object and the data write amount of the current cycle; determine the release bandwidth of the current cycle according to the redundancy type, redundancy quantity of each object and the garbage release amount of the current cycle; determine the write bandwidth of the previous cycle according to the redundancy type, redundancy quantity of each object and the data write amount of the previous cycle; determine the release bandwidth of the previous cycle according to the redundancy type, redundancy quantity of each object and the garbage release amount of the previous cycle.

[0028] In this embodiment, the redundancy type and redundancy quantity of each object can be determined first. Here, the redundancy type may include replicas and erasure code redundancy. A replica refers to the redundant storage of data, that is, multiple independent copies of the data are stored on different servers. The purpose of this method is to ensure the reliability and availability of the data. Even if some servers fail, other copies of the data can still be accessed and used. Erasure code redundancy divides the original data into multiple data blocks and encodes these data blocks to generate redundant data blocks (check blocks) to provide data fault tolerance.

[0029] After determining the redundancy type and redundancy quantity, the write bandwidth and release bandwidth can be determined in combination with the data write amount and garbage release amount. Specifically, the write bandwidth of the current cycle can be determined according to the redundancy type, redundancy quantity of each object and the data write amount of the current cycle. If the redundancy type of each object is a replica and the number of replicas is 3. If the data write amount of this object to the hard disk is 3T, then the actual write bandwidth is 3T / 3 = 1T. If the redundancy type of each object is erasure code redundancy and the redundancy quantity is 4 original data elements and 2 redundant elements. If the data write amount of this object to the hard disk is 3T, then the actual write bandwidth is 3T * 4 / (4 + 2) = 2T.

[0030] Similarly, the release bandwidth of the current cycle can be determined based on the redundancy type, redundancy quantity of each object, and the garbage release amount of the current cycle; the write bandwidth of the previous cycle can be determined based on the redundancy type, redundancy quantity of each object, and the data write amount of the previous cycle; the release bandwidth of the previous cycle can be determined based on the redundancy type, redundancy quantity of each object, and the garbage release amount of the previous cycle.

[0031] Step 203: Determine the first difference between the write bandwidth of the current cycle and the write bandwidth of the previous cycle; determine the second difference between the release bandwidth of the current cycle and the release bandwidth of the previous cycle; dynamically adjust the garbage collection parameters according to the first difference and the second difference.

[0032] After calculating the write bandwidth of the current cycle, the write bandwidth of the previous cycle, the release bandwidth of the current cycle, and the release bandwidth of the previous cycle, the first difference between the write bandwidth of the current cycle and the write bandwidth of the previous cycle can be calculated, and at the same time, the second difference between the release bandwidth of the current cycle and the release bandwidth of the previous cycle can be calculated. And further dynamically adjust the garbage collection parameters according to the first difference and the second difference.

[0033] Specifically, if the first difference and the second difference are close to 0, it means that the write bandwidth of the current cycle is equal to the write bandwidth of the previous cycle, and the release bandwidth of the current cycle is equal to the release bandwidth of the previous cycle, and the current state is in a balanced state, then there is no need to adjust the garbage collection parameters. If the first difference is less than the first preset threshold, and the second difference is greater than the second preset threshold, it means that garbage collection in the current state occupies too many resources, and the garbage collection parameters need to be reduced to reduce the resources occupied by garbage collection.

[0034] In some optional implementation manners of this embodiment, if the first difference is less than the first preset threshold and the second difference is greater than the second preset threshold, the garbage collection parameters can be reduced. If the error between the first difference and the preset value is less than the third preset threshold and the second difference is less than the fourth preset threshold, the garbage collection parameters can be increased.

[0035] In this implementation manner, if the first difference is less than the first preset threshold, and the second difference is greater than the second preset threshold, it means that garbage collection in the current state occupies too many resources, and the garbage collection parameters need to be reduced to reduce the resources occupied by garbage collection. If the first difference is equivalent to the preset value (here, the preset value can be 0), and the second difference is less than the fourth preset threshold, it means that the resources occupied by the current garbage collection are less, and the garbage collection parameters can be increased.

[0036] In some alternative implementation manners of this embodiment, if the relative error between the write bandwidth in the current period and the release bandwidth in the current period is less than a fifth preset threshold (the fifth preset threshold may be 0), that is, the data write amount in the current period is equivalent to the garbage release amount, the garbage collection parameters may be kept unchanged.

[0037] Step 204: According to the garbage collection parameters, perform reverse lookup on the metadata of each object to determine whether each object is valid data; in response to determining that each object is valid data, move the object to another location and modify the metadata of the object; in response to determining that each object is invalid data, release the storage space occupied by the object.

[0038] After adjusting the garbage collection parameters, the metadata of each object may be reverse looked up according to the garbage collection parameters to determine whether each object is valid data. If it is determined that the object is valid data, the object may be moved to another location. At the same time, the metadata of the object is modified. Specifically, the new storage address of the object is stored in the updated metadata. If the object is invalid data, the storage space occupied by the object may be released.

[0039] In some alternative implementation manners of this embodiment, the period duration and the start time of the period set by the user may also be obtained. According to the above period duration and the start time of the period, the start time and the end time of data acquisition for each period are determined. For example, the period duration set by the user is 30 minutes and the start time is 0:00. Then data can be acquired starting from 0:00 and the data acquisition for one period ends at 0:30. The start time of the next period is 0:31 and the end time is 1:00.

[0040] The hard disk garbage collection method provided in the above embodiments of the present disclosure can automatically calculate and compare the write bandwidth in the foreground and the release bandwidth in the background, adaptively adjust the garbage collection parameters to achieve the balance between the two, can cope with the fluctuations and changes of the write bandwidth in the foreground, ensure the continuous stability of the cluster capacity, improve the robustness of the system, and at the same time reduce the manual maintenance and interference costs.

[0041] Further referring to Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a hard disk garbage collection device. This device embodiment corresponds to Figure 1 the method embodiment shown, and this device can be specifically applied to various electronic devices.

[0042] As Figure 3 shown, the hard disk garbage collection device 300 in this embodiment includes: an acquisition unit 301, a determination unit 302, an adjustment unit 303, and a recovery unit 304.

[0043] An acquisition unit 301, configured to acquire the data write amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle.

[0044] A determination unit 302, configured to determine the write bandwidth in the current cycle, the release bandwidth in the current cycle, the write bandwidth in the previous cycle, and the release bandwidth in the previous cycle according to the redundancy quantity of each object, the data write amount and garbage release amount in the current cycle and the previous cycle.

[0045] An adjustment unit 303, configured to dynamically adjust the garbage collection parameters according to the write bandwidth in the current cycle, the release bandwidth in the current cycle, the write bandwidth in the previous cycle, and the release bandwidth in the previous cycle.

[0046] A recycling unit 304, configured to perform garbage collection on each storage unit of the hard disk according to the garbage collection parameters.

[0047] In addition, in the technical solution of this application, an electronic device is also proposed.

[0048] Figure 4 The structural schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0049] As Figure 4 shown, the electronic device may include a processor 401, a memory 402, a bus 403, and a computer program stored on the memory 402 and executable on the processor 401. Among them, the processor 401 and the memory 402 complete communication with each other through the bus 403. When the processor 401 executes the computer program, the steps of the above method are implemented, for example, including: acquiring the data write amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle; determining the write bandwidth in the current cycle, the release bandwidth in the current cycle, the write bandwidth in the previous cycle, and the release bandwidth in the previous cycle according to the redundancy quantity of each object, the data write amount and garbage release amount in the current cycle and the previous cycle; dynamically adjusting the garbage collection parameters according to the write bandwidth in the current cycle, the release bandwidth in the current cycle, the write bandwidth in the previous cycle, and the release bandwidth in the previous cycle; performing garbage collection on each storage unit of the hard disk according to the garbage collection parameters.

[0050] In addition, in an embodiment of the present disclosure, a non-transitory computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented, for example, including: obtaining the data write amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle; determining the write bandwidth in the current cycle, the release bandwidth in the current cycle, the write bandwidth in the previous cycle, and the release bandwidth in the previous cycle according to the redundancy quantity of each object, the data write amount and garbage release amount in the current cycle and the previous cycle; dynamically adjusting the garbage collection parameters according to the write bandwidth in the current cycle, the release bandwidth in the current cycle, the write bandwidth in the previous cycle, and the release bandwidth in the previous cycle; and performing garbage collection on each storage unit of the hard disk according to the garbage collection parameters.

[0051] In summary, in the technical solution of the present disclosure, the corresponding write bandwidth and release bandwidth can be determined according to the data write amount and garbage release amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle, and the garbage collection parameters are dynamically adjusted according to the corresponding write bandwidth and release bandwidth. Further, garbage collection is performed on the hard disk according to the garbage collection parameters, so as to be able to cope with the fluctuations and changes of the foreground read / write bandwidth and improve the robustness of the system.

[0052] The foregoing is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A hard disk garbage collection method, comprising: Get the amount of data written and garbage released for each object in each storage unit of the hard disk in the current cycle and the previous cycle; Determine the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle according to the redundant quantity of each object, the data write amount and the garbage release amount of the current cycle and the previous cycle; Dynamically adjusting garbage collection parameters according to the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle; According to the garbage collection parameters, garbage collection is performed on each storage unit of the hard disk.

2. The method according to claim 1, wherein: According to the redundancy number of each object, the data written amount and garbage released amount in the current cycle and the previous cycle, the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle and the release bandwidth of the previous cycle are determined, including: Determine the redundancy type and amount of each object; Determine the write bandwidth of the current cycle according to the redundancy type and redundancy quantity of each object and the data write volume of the current cycle; Determine the release bandwidth of the current cycle according to the redundancy type and redundancy quantity of each object and the garbage release amount of the current cycle; Determine the write bandwidth of the previous cycle according to the redundancy type and redundancy quantity of each object and the data written amount of the previous cycle; The release bandwidth of the previous cycle is determined according to the redundancy type and redundancy quantity of each object and the garbage release amount of the previous cycle.

3. The method according to claim 1, wherein: The dynamically adjusting the garbage collection parameters according to the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle includes: Determine a first difference between a write bandwidth of a current cycle and a write bandwidth of a previous cycle; Determine a second difference between a released bandwidth of a current cycle and a released bandwidth of a previous cycle; The garbage collection parameter is dynamically adjusted according to the first difference and the second difference.

4. The method according to claim 3, wherein: The dynamically adjusting the garbage collection parameter according to the first difference and the second difference includes: In response to determining that the first difference is less than a first preset threshold and the second difference is greater than a second preset threshold, reducing the garbage collection parameter; In response to determining that the error between the first difference and the preset value is less than a third preset threshold and the second difference is less than a fourth preset threshold, the garbage collection parameter is increased.

5. The method according to claim 4, wherein: The dynamically adjusting the garbage collection parameter according to the first difference and the second difference includes: In response to determining that a relative error between the write bandwidth of the current cycle and the release bandwidth of the current cycle is less than a fifth preset threshold, the garbage collection parameter is kept unchanged.

6. The method according to claim 1, wherein: The performing garbage collection on each storage unit of the hard disk according to the garbage collection parameter includes: According to the garbage collection parameters, the metadata of each object is checked to determine whether each object is valid data; In response to determining that each object is valid data, moving the object to another location and modifying metadata of the object; In response to determining that each object is invalid data, the storage space occupied by the object is released.

7. The method according to claim 1, wherein: The method further comprises: Get the cycle duration and cycle start time set by the user; The start time and end time of data acquisition of each cycle are determined according to the cycle duration and the cycle start time.

8. A hard disk garbage collection device, comprising: An acquisition unit is configured to acquire the data writing amount and garbage releasing amount of each object in each storage unit of the hard disk in the current cycle and the previous cycle; A determination unit is configured to determine a write bandwidth of a current cycle, a release bandwidth of a current cycle, a write bandwidth of a previous cycle, and a release bandwidth of a previous cycle according to the redundancy quantity of each object, the data write quantity and the garbage release quantity of the current cycle and the previous cycle; an adjustment unit, configured to dynamically adjust garbage collection parameters according to the write bandwidth of the current cycle, the release bandwidth of the current cycle, the write bandwidth of the previous cycle, and the release bandwidth of the previous cycle; The recycling unit is configured to perform garbage collection on each storage unit of the hard disk according to the garbage collection parameters.

9. An electronic device comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the hard disk garbage collection method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the hard disk garbage collection method as described in any one of claims 1 to 7 is implemented.

Images