Hard disk garbage collection method and device, electronic equipment and storage medium

Abstract

The invention relates to a hard disk garbage collection method and device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: for each storage unit in the hard disk, dividing each garbage collection object in the storage unit to obtain a plurality of small storage blocks; for each small storage block, determining the proportion of invalid data in the small storage block; determining a first garbage collection threshold value corresponding to each garbage collection object according to the proportion of invalid data in each small storage block; determining a second garbage collection threshold value corresponding to each storage unit according to the first garbage collection threshold value corresponding to each garbage collection object; and performing garbage collection on each small storage block according to the second garbage collection threshold value. By applying the method and the device, the recovery efficiency can be improved while the moving amount is reduced.

Description

Technical Field

[0001] This application relates to the field of computer technologies, and in particular, to a hard disk garbage collection method, apparatus, electronic device, and storage medium. Background Art

[0002] To better utilize the write performance of a hard disk, a common practice in the industry is to redirect the data written to the hard disk pool object during write (Redirect-On-Write, ROW) together and aggregate it into large I / O (input / output), and then sequentially write it to the object allocated by the hard disk pool. In the field of distributed storage, by adopting the append-write method, the write performance of data can be greatly improved, so ROW also uses this technology. However, when appending the same piece of data, corresponding garbage data will be generated. Therefore, the industry usually uses a garbage collection (GC) algorithm to recycle the corresponding garbage space. Since ROW generally uses the technology of large objects (such as an object of 128M), the general approach is to scan these objects and perform garbage collection and relocation on the objects that meet the threshold of the garbage amount, so as to achieve the purpose of releasing space. However, this solution does not consider the overall distribution of cluster garbage. For the scenario where random write garbage is relatively scattered, performing garbage collection on large objects will result in a large amount of relocation, not only with small benefits but also possibly affecting the foreground service. Summary of the Invention

[0003] Embodiments of the present disclosure provide a hard disk garbage collection method, apparatus, electronic device, and storage medium.

[0004] In a first aspect, embodiments of the present disclosure provide a hard disk garbage collection method, including: for each storage unit in the hard disk, dividing each garbage collection object in the storage unit to obtain a plurality of small storage blocks; for each small storage block, determining the proportion of invalid data in the small storage block; according to the proportion of invalid data in each small storage block, determining a first garbage collection threshold corresponding to each garbage collection object; according to the first garbage collection threshold corresponding to each garbage collection object, determining a second garbage collection threshold corresponding to each storage unit; and performing garbage collection on each small storage block according to the second garbage collection threshold.

[0005] In a second aspect, an embodiment of the present disclosure provides a hard disk garbage collection device, including: an object division unit configured to divide each garbage collection object in each storage unit in the hard disk to obtain a plurality of small storage blocks; a proportion determination unit configured to determine, for each small storage block, the proportion of invalid data in the small storage block; a first threshold determination unit configured to determine, according to the proportion of invalid data in each small storage block, a first garbage collection threshold corresponding to each garbage collection object; a second threshold determination unit configured to determine, according to the first garbage collection threshold corresponding to each garbage collection object, a second garbage collection threshold corresponding to each storage unit; and a garbage collection unit configured to perform garbage collection on each small storage block according to the second garbage collection threshold.

[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device, including a memory, a processor, a bus, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the hard disk garbage collection method described in the first aspect is implemented.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the hard disk garbage collection method described in the first aspect is implemented.

[0008] By applying the technical solution of the present disclosure, each garbage collection object in each storage unit in the hard disk can be divided to obtain a plurality of small storage blocks; then, the proportion of garbage data in each small storage block is calculated. And according to the above proportion, a first garbage collection threshold corresponding to each redundancy type is determined, and further a second garbage collection threshold corresponding to different redundancy types of each storage unit is determined. Finally, garbage collection is performed on each small storage block according to the second garbage collection threshold. Thereby, while reducing the amount of relocation, the recovery efficiency can be improved.

[0009] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them: Figure 1 is a flowchart of an embodiment of the hard disk garbage collection method of the present disclosure; Figure 2 is a flowchart of another embodiment of the hard disk garbage collection method of the present disclosure; Figure 3 shows the correspondence between the proportion of invalid data in the small storage block and the number of small storage blocks; Figure 4 Shows a schematic diagram of the process for determining the second garbage collection threshold in the hard disk garbage collection method of the present disclosure; Figure 5 Is a schematic structural diagram of an embodiment of the hard disk garbage collection device of the present disclosure; Figure 6 Is a schematic structural diagram of an embodiment of the electronic device of the present disclosure. Detailed implementation manners

[0011] It should be noted that the following detailed descriptions are all exemplary and are intended to provide further descriptions of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present disclosure belongs.

[0012] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary implementation manners according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0013] In the case of no conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other.

[0014] To make the technical solutions and advantages of the present disclosure clearer and more understandable, the present disclosure will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] Figure 1 Shows a flow 100 of an embodiment of the hard disk garbage collection method of the present disclosure. As Figure 1 shown, the hard disk garbage collection method of this embodiment may include the following steps: Step 101, for each storage unit in the hard disk, divide each garbage collection object in the storage unit to obtain a plurality of small storage blocks.

[0016] In this embodiment, garbage collection can be performed on each storage unit in the hard disk. Here, the storage unit can be the basic unit of the hard disk, such as rank, bank, etc. Various objects can be stored in each storage unit. Each type of object can include various types of redundancy, such as replicas, erasure code redundancy, etc.

[0017] Each storage unit can include a plurality of garbage collection objects, and the size of each garbage collection object here can be 128M. To improve the efficiency of garbage collection, the garbage collection objects can be divided in this embodiment.

[0018] During partitioning, it can be done according to a preset size. For example, if the preset size is 4M, then the garbage collection object can be divided into 128 / 4 = 32 blocks. The size of each small storage block is 4M. Or, it can be divided according to the number of copies of the objects stored in the garbage collection object.

[0019] Step 102: For each small storage block, determine the proportion of invalid data in it.

[0020] After obtaining each small storage block by partitioning, the data in the small storage block can be identified to determine the valid data and invalid data among them. Then, according to the volume of the invalid data and the size of each small storage block, determine the proportion of invalid data in each small storage block.

[0021] Step 103: According to the proportion of invalid data in each small storage block, determine the first garbage collection threshold corresponding to each garbage collection object.

[0022] After determining the proportion of invalid data in each small storage block, the first garbage collection threshold corresponding to each garbage collection object can be further determined. Specifically, a proportion threshold can be determined first. Then divide the number of small storage blocks with a proportion greater than this threshold by the total number of all small storage blocks, and use the obtained ratio as the first garbage collection threshold. Or, a quantity threshold can be determined first. Starting from the highest proportion, add up the quantities of the small storage blocks corresponding to each proportion in descending order until the sum first exceeds this quantity threshold and then stop. The corresponding proportion value at this time is used as the first garbage collection threshold.

[0023] Step 104: According to the first garbage collection threshold corresponding to each garbage collection object, determine the second garbage collection threshold corresponding to each storage unit.

[0024] After determining the first garbage collection threshold corresponding to each garbage collection object, the second garbage collection threshold corresponding to each storage unit can be further determined. Specifically, the first garbage collection thresholds corresponding to the garbage collection objects belonging to the same storage unit can be weighted, and the obtained value is used as the second garbage collection threshold corresponding to this storage unit. The weights here can be preset or determined according to the number of small storage blocks included in each garbage collection object.

[0025] Step 105: Perform garbage collection on each small storage block according to the second garbage collection threshold.

[0026] After determining the second garbage collection threshold corresponding to each storage unit, garbage collection can be performed on each small storage block according to the second garbage collection threshold. Specifically, the data in each small storage block can be identified to determine valid data and invalid data. If the proportion of invalid data exceeds the second garbage collection threshold, garbage collection can be performed on each small storage block.

[0027] The hard disk garbage collection method provided by the above embodiments of the present disclosure splits the garbage collection object into small granularities compared with the existing garbage collection methods, and at the same time determines a suitable garbage collection threshold based on statistics, so as to improve the efficiency of garbage collection and effectively reduce the risk of the data space being filled up.

[0028] Continue to refer to Figure 2 , which shows the flow 200 of another embodiment of the hard disk garbage collection method according to the present disclosure. As Figure 2 shown, the method in this embodiment may include the following steps: Step 201, for each redundancy type corresponding to a single garbage collection object in each storage unit in the hard disk, determine the size of the small storage block corresponding to the redundancy type.

[0029] In this embodiment, when dividing a single garbage collection object in each storage unit in the hard disk, the redundancy type corresponding to each garbage collection object can be considered. Here, the redundancy type may include but is not limited to: replicas, erasure code (abbreviated as EC) redundancy. A replica refers to the redundant storage of data, that is, multiple independent copies of the data are stored on different servers. The purpose of this method is to ensure the reliability and availability of the data. Even if some servers fail, other copies of the data can still be accessed and used. Erasure code redundancy is a data redundancy technology that slices the original data into multiple data blocks and encodes these data blocks to generate redundant data blocks (check blocks) to provide data fault tolerance.

[0030] The size of the corresponding small storage block can be determined according to the redundancy type. Specifically, the size of the corresponding small storage block can be determined according to the number of data slices corresponding to the redundancy type. For example, when the redundancy type is N replicas, the size of the small storage block can be set to a preset value or a power of 2 value closest to N. Taking N = 3 as an example, the size of the small storage block can be set to 4M. The erasure code type is calculated according to the data slices * 1M. For example, for EC2+1 and EC2+2, the number of data slices is 2, and it is calculated as 2M. For EC4+1 and EC4+2, the number of data slices is 4, and it is calculated as 4M. For EC8+1, EC8+2, etc., it is similar.

[0031] Step 202: For each garbage collection object, divide the garbage collection object according to the sizes of the small storage blocks corresponding to each redundancy type of the garbage collection object, to obtain a plurality of small storage blocks.

[0032] After determining the sizes of the small storage blocks corresponding to each redundancy type, each garbage collection object can be divided according to the size to obtain a plurality of small storage blocks. It can be understood that the sizes of the small storage blocks corresponding to different redundancy types are different. When dividing the garbage collection object, multiple division schemes can be obtained. The number of division schemes is the same as the value of the small storage blocks.

[0033] Step 203: For each small storage block, determine the proportion of invalid data in the small storage block.

[0034] Step 204: For each redundancy type corresponding to a single garbage collection object, according to the proportion of invalid data in each small storage block divided according to the redundancy type, determine the number of small storage blocks corresponding to each proportion; according to the number of small storage blocks corresponding to each proportion, determine the first garbage collection threshold corresponding to different redundancy types of a single garbage collection object.

[0035] After determining the proportion of garbage data in each small storage block, the number of small storage blocks corresponding to each proportion value can be determined. Specifically, a series of small storage blocks divided according to a single redundancy type can be analyzed. That is, for redundancy type A, the size of its corresponding small storage block is M. N small storage blocks are obtained according to M. The number of small storage blocks corresponding to a single proportion can be counted according to the proportion of invalid data in the N small storage blocks. Then, according to the number of small storage blocks corresponding to each proportion, determine the first garbage collection threshold corresponding to different redundancy types of a single garbage collection object. Specifically, the proportion corresponding to the highest value can be used as the first garbage collection threshold.

[0036] In some alternative implementation manners of this embodiment, a curve can be drawn with the proportion as the abscissa and the number of small storage blocks as the ordinate. It can be understood that the curves corresponding to different redundancy types are different. Then, according to the curve and a preset threshold, determine the first garbage collection threshold. The preset threshold can be understood as the area value between the curve, the abscissa axis, and the vertical line corresponding to the first garbage collection threshold, and can also be understood as the ratio of the above area value to the total area of the region surrounded by the curve and the two coordinate axes.

[0037] Combined with Figure 3 , Figure 3The corresponding relationship between the proportion of invalid data in small storage blocks and the number of small storage blocks is shown. The preset threshold can be set to 1 / 4, indicating the ratio of the area covered by the vertical line to the area between the curve and the two coordinate axes. Through this ratio, the abscissa value corresponding to the leftmost vertical line is determined as the first garbage collection threshold.

[0038] Step 205: For each redundancy type, weight the first garbage collection thresholds corresponding to this redundancy type to determine the second garbage collection thresholds corresponding to different redundancy types of each storage unit.

[0039] After determining the first garbage collection thresholds corresponding to the redundancy types of a single garbage collection object, further statistics can be performed on each garbage collection object. Specifically, the first garbage collection thresholds corresponding to each garbage collection object in a single storage unit can be weighted according to the redundancy type to determine the second garbage collection thresholds corresponding to different redundancy types of a single storage unit.

[0040] When determining the weights of the first garbage collection thresholds, it can be determined according to the number of small storage blocks included in each garbage collection object. The more small storage blocks included in a garbage collection object, the greater the weight of this first garbage collection threshold.

[0041] Step 206: For each garbage collection object, in response to determining that the proportion of invalid data in this garbage collection object is greater than the second garbage collection threshold, perform garbage collection on the small storage blocks in this garbage collection object whose proportion of invalid data is greater than the second garbage collection threshold.

[0042] After determining the second garbage collection thresholds corresponding to the redundancy types, garbage collection can be performed on each small storage block. Specifically, it can first be determined whether the proportion of invalid data in each garbage collection object is greater than the second garbage collection threshold. It can be understood that if a single garbage collection object corresponds to multiple redundancy types and the second garbage collection thresholds corresponding to each redundancy type are different, as long as the proportion of invalid data in a single garbage collection object is greater than the minimum value among the second garbage collection thresholds, further judgment can be made on the small storage blocks in this garbage collection object. That is, perform garbage collection on the small storage blocks in this garbage collection object whose proportion of invalid data is greater than the second garbage collection threshold. In some specific practices, if the data stored in a single small storage block includes multiple redundancy types and the second garbage collection thresholds corresponding to each redundancy type are different. When performing garbage collection on this small storage block, the proportion of invalid data can be compared with the minimum value among the second garbage collection thresholds.

[0043] In some alternative implementation manners of this embodiment, when performing garbage collection on a small storage block, the valid data in the small storage block can be migrated first, migrated to other free storage units, and then the invalid data in the small storage block can be deleted, so as to release the space of the small storage block.

[0044] Combined with Figure 4 for understanding, Figure 4 FIG. shows a schematic diagram of the determination process of the second garbage collection threshold in the hard disk garbage collection method of the present disclosure. The producer performs garbage scanning on a single storage unit rank of the hard disk to obtain a plurality of garbage collection objects tgt0 to tgtN. For each garbage collection object tgt, after being divided according to the redundancy type (redundancy 1 to redundancy N shown in the figure), a plurality of small storage blocks are obtained. Determine the proportion of invalid data in each small storage block. The proportions are respectively counted according to 0 to 100 for the number of small storage blocks to obtain the proportions corresponding to different redundancy types. According to the recovery threshold determination rule, that is, the preset threshold is 1 / 4, the first garbage collection threshold TWL corresponding to each redundancy type is determined. Then, the first garbage collection thresholds TWL are weighted to obtain the second recovery threshold RWL.

[0045] The hard disk garbage collection method provided by the above embodiment of the present disclosure can split a large garbage collection object into small grains to obtain a plurality of small storage blocks. And the proportion of invalid data in each small storage block is counted to obtain the optimal garbage collection threshold, thereby reducing the amount of valid data migration during the garbage collection process, improving the garbage collection efficiency, and effectively reducing the risk of the data space being filled up.

[0046] Further referring to Figure 5 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a hard disk garbage collection device. This device embodiment corresponds to Figure 1 the method embodiment shown, and this device can be specifically applied to various electronic devices.

[0047] As Figure 5 shown, the hard disk garbage collection device 500 of this embodiment includes: an object division unit 501, a proportion determination unit 502, a first threshold determination unit 503, a second threshold determination unit 504, and a garbage collection unit 505.

[0048] The object division unit 501 is configured to divide each garbage collection object in each storage unit of the hard disk to obtain a plurality of small storage blocks.

[0049] The proportion determination unit 502 is configured to determine the proportion of invalid data in each small storage block.

[0050] The first threshold determination unit 503 is configured to determine the first garbage collection threshold corresponding to each garbage collection object according to the proportion of invalid data in each small storage block.

[0051] The second threshold determination unit 504 is configured to determine the second garbage collection threshold corresponding to each storage unit according to the first garbage collection threshold corresponding to each garbage collection object.

[0052] The garbage collection unit 505 is configured to perform garbage collection on each small storage block according to the second garbage collection threshold.

[0053] In addition, in the technical solution of this application, an electronic device is also proposed.

[0054] Figure 6 The structural schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0055] As Figure 6 shown, the electronic device may include a processor 601, a memory 602, a bus 603, and a computer program stored on the memory 602 and executable on the processor 601. Among them, the processor 601 and the memory 602 complete communication with each other through the bus 603. When the processor 601 executes the computer program, the steps of the above method are implemented, for example, including: for each storage unit in the hard disk, dividing each garbage collection object in the storage unit to obtain a plurality of small storage blocks; for each small storage block, determining the proportion of invalid data in the small storage block; according to the proportion of invalid data in each small storage block, determining the first garbage collection threshold corresponding to each garbage collection object; according to the first garbage collection threshold corresponding to each garbage collection object, determining the second garbage collection threshold corresponding to each storage unit; and performing garbage collection on each small storage block according to the second garbage collection threshold.

[0056] In addition, in an embodiment of the present disclosure, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented, for example, including: for each storage unit in the hard disk, dividing each garbage collection object in the storage unit to obtain a plurality of small storage blocks; for each small storage block, determining the proportion of invalid data in the small storage block; according to the proportion of invalid data in each small storage block, determining the first garbage collection threshold corresponding to each garbage collection object; according to the first garbage collection threshold corresponding to each garbage collection object, determining the second garbage collection threshold corresponding to each storage unit; and performing garbage collection on each small storage block according to the second garbage collection threshold.

[0057] In summary, in the technical solution of the present disclosure, each garbage collection object in each storage unit in the hard disk can be divided to obtain a plurality of small storage blocks; then, the proportion of garbage data in each small storage block is calculated. And the first garbage collection threshold corresponding to each redundancy type is determined according to the above proportion, and further the second garbage collection threshold corresponding to different redundancy types of each storage unit is determined. Finally, garbage collection is performed on each small storage block according to the second garbage collection threshold. Thereby, while reducing the amount of migration, the recovery efficiency can be improved.

[0058] The foregoing are only the preferred embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A hard disk garbage collection method, comprising: For each storage unit in the hard disk, each garbage collection object in the storage unit is divided to obtain multiple small storage blocks; For each small storage block, determine the proportion of invalid data in the small storage block; Determine a first garbage collection threshold corresponding to each garbage collection object according to the proportion of invalid data in each small storage block; Determine a second garbage collection threshold corresponding to each storage unit according to the first garbage collection threshold corresponding to each garbage collection object; Garbage collection is performed on each small storage block according to the second garbage collection threshold.

2. The method according to claim 1, wherein: Each garbage collection object corresponds to at least one redundancy type; as well as The garbage collection objects in the storage unit are divided into multiple small storage blocks, including: For each redundancy type corresponding to a single garbage collection object, determine the size of a small storage block corresponding to the redundancy type; For each garbage collection object, the garbage collection object is divided according to the sizes of small storage blocks corresponding to each redundancy type corresponding to the garbage collection object to obtain a plurality of small storage blocks.

3. The method according to claim 2, wherein: The determining, according to the proportion of invalid data in each small storage block, a first garbage collection threshold corresponding to each garbage collection object includes: For each redundancy type corresponding to a single garbage collection object, according to the proportion of garbage data in each small storage block divided according to the redundancy type, determine the number of small storage blocks corresponding to each proportion; A first garbage collection threshold corresponding to different redundancy types corresponding to a single garbage collection object is determined according to the number of small storage blocks corresponding to each proportion.

4. The method according to claim 3, wherein: The determining, according to the number of small storage blocks corresponding to each proportion, first garbage collection thresholds corresponding to different redundancy types corresponding to a single garbage collection object includes: For each redundancy type, the ratio of garbage data in each small storage block divided according to the redundancy type is used as the horizontal coordinate, and the number of small storage blocks corresponding to the ratio is used as the vertical coordinate to draw a curve corresponding to the redundancy type; According to the curve and the preset threshold, a first garbage collection threshold corresponding to different redundancy types of a single garbage collection object is determined.

5. The method according to claim 4, wherein: The determining, according to the first garbage collection threshold corresponding to each garbage collection object, the second garbage collection threshold corresponding to each storage unit includes: For each redundancy type, each first garbage collection threshold corresponding to the redundancy type is weighted to determine a second garbage collection threshold corresponding to different redundancy types of each storage unit.

6. The method according to claim 1, wherein: The performing garbage collection on each small storage block according to the second garbage collection threshold comprises: For each garbage collection object, in response to determining that the proportion of invalid data in the garbage collection object is greater than the second garbage collection threshold, garbage collection is performed on small storage blocks in which the proportion of invalid data in the garbage collection object is greater than the second garbage collection threshold.

7. The method according to claim 6, wherein: The performing garbage collection on the small storage block whose proportion of invalid data in the garbage collection object is greater than the second garbage collection threshold comprises: The valid data in the small storage blocks in which the proportion of invalid data in the garbage collection object is greater than the second garbage collection threshold is migrated, and the invalid data is deleted.

8. A hard disk garbage collection device, comprising: The object partitioning unit is configured to partition each garbage collection object in each storage unit in the hard disk to obtain a plurality of small storage blocks; A proportion determination unit is configured to determine, for each small storage block, a proportion of invalid data in the small storage block; A first threshold determination unit is configured to determine a first garbage collection threshold corresponding to each garbage collection object according to a proportion of invalid data in each small storage block; A second threshold determination unit is configured to determine a second garbage collection threshold corresponding to each storage unit according to the first garbage collection threshold corresponding to each garbage collection object; The garbage collection unit is configured to perform garbage collection on each small storage block according to the second garbage collection threshold.

9. An electronic device comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the hard disk garbage collection method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the hard disk garbage collection method as described in any one of claims 1 to 7 is implemented.

Images