Hard Disk Garbage Collection Method, Device, Electronic Device and Storage Medium

Abstract

The present disclosure relates to a hard disk garbage collection method, device, electronic device and storage medium, and relates to the field of computer technology. The method includes: for each storage unit in the hard disk, dividing each garbage collection object in the storage unit to obtain multiple small storage blocks; for each small storage block, determining the proportion of invalid data in the small storage block; determining the first garbage collection threshold corresponding to each garbage collection object based on the proportion of invalid data in each small storage block; determining the second garbage collection threshold corresponding to each storage unit based on the first garbage collection threshold corresponding to each garbage collection object; and performing garbage collection on each small storage block based on the second garbage collection threshold. The application of the present disclosure can improve the recycling efficiency while reducing the amount of movement.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a hard disk garbage recovery method, device, electronic device, and storage medium. Background Art

[0002] To maximize disk write performance, a common industry practice is to redirect data written to disk pool objects (Redirect-On-Write (ROW)) and aggregate it into large I / Os (input / output), which are then written sequentially to objects allocated by the disk pool. In the distributed storage field, appending to the same block of data can significantly improve data write performance, so ROW also adopts this technique. However, appending to the same block of data generates garbage data, so the industry typically uses garbage collection (GC) algorithms to reclaim the corresponding garbage space. Since ROW typically uses large objects (such as 128MB objects), the common approach is to scan these objects and garbage collect and move objects whose garbage volume meets a certain threshold to free up space. However, this solution does not consider the overall distribution of garbage in the cluster. In scenarios where random writes are relatively dispersed, garbage collection of large objects results in a large amount of garbage moved, which not only yields little benefit but also potentially impacts front-end business operations. Summary of the Invention

[0003] Embodiments of the present disclosure provide a hard disk garbage collection method, apparatus, electronic device, and storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a hard disk garbage collection method, comprising: for each storage unit in the hard disk, dividing each garbage collection object in the storage unit to obtain multiple small storage blocks; for each small storage block, determining the proportion of invalid data in the small storage block; determining a first garbage collection threshold corresponding to each garbage collection object based on the proportion of invalid data in each small storage block; determining a second garbage collection threshold corresponding to each storage unit based on the first garbage collection threshold corresponding to each garbage collection object; and performing garbage collection on each small storage block based on the second garbage collection threshold.

[0005] In the second aspect, an embodiment of the present disclosure provides a hard disk garbage collection device, comprising: an object division unit, configured to divide each garbage collection object in each storage unit in the hard disk to obtain multiple small storage blocks; a proportion determination unit, configured to determine the proportion of invalid data in each small storage block; a first threshold determination unit, configured to determine the first garbage collection threshold corresponding to each garbage collection object according to the proportion of invalid data in each small storage block; a second threshold determination unit, configured to determine the second garbage collection threshold corresponding to each storage unit according to the first garbage collection threshold corresponding to each garbage collection object; and a garbage collection unit, configured to perform garbage collection on each small storage block according to the second garbage collection threshold.

[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the hard disk garbage collection method as described in the first aspect is implemented.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hard disk garbage collection method as described in the first aspect.

[0008] Applying the technical solution disclosed herein, each garbage collection pair within each storage unit on a hard drive can be divided into multiple small storage blocks. The percentage of garbage data in each small storage block is then calculated. Based on this percentage, a first garbage collection threshold corresponding to each redundancy type is determined. Furthermore, a second garbage collection threshold corresponding to each redundancy type in each storage unit is determined. Finally, garbage collection is performed on each small storage block based on the second garbage collection threshold. This reduces the amount of data moved while improving recycling efficiency.

[0009] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0011] Figure 1 A flowchart of an embodiment of the hard disk garbage collection method disclosed herein;

[0012] Figure 2 This is a flow chart of another embodiment of the hard disk garbage collection method disclosed herein;

[0013] Figure 3 The corresponding relationship between the proportion of invalid data in the small storage blocks and the number of small storage blocks is shown;

[0014] Figure 4 A schematic diagram showing a process for determining a second garbage collection threshold value in the hard disk garbage collection method disclosed herein is shown;

[0015] Figure 5 This is a structural diagram of an embodiment of a hard disk garbage collection device disclosed herein;

[0016] Figure 6 This is a schematic structural diagram of an embodiment of an electronic device disclosed herein. DETAILED DESCRIPTION

[0017] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0018] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0019] In the absence of conflict, the embodiments of the present disclosure and the features thereof may be combined with each other.

[0020] In order to make the technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Figure 1 FIG. 1 shows a process 100 of an embodiment of the hard disk garbage collection method disclosed herein. Figure 1 As shown, the hard disk garbage collection method of this embodiment may include the following steps:

[0022] Step 101 : For each storage unit in the hard disk, each garbage collection object in the storage unit is divided to obtain a plurality of small storage blocks.

[0023] In this embodiment, garbage collection can be performed on each storage unit in the hard disk. Here, the storage unit can be a basic unit of the hard disk, such as a rank or bank. Various objects can be stored in each storage unit. Each object can include multiple types of redundancy, such as replicas, erasure code redundancy, and so on.

[0024] Each storage unit may include multiple garbage collection objects, where the size of each garbage collection object may be 128 M. In order to improve the efficiency of garbage collection, in this embodiment, each garbage collection object may be divided.

[0025] When partitioning, you can use a preset size. For example, if the preset size is 4MB, you can partition the garbage collection object into 128 / 4 = 32 blocks. Each small storage block is 4MB in size. Alternatively, you can partition based on the number of replicas of the object stored in the garbage collection object.

[0026] Step 102: For each small storage block, determine the proportion of invalid data in the small storage block.

[0027] After the small storage blocks are divided, the data in the small storage blocks can be identified to determine the valid data and invalid data therein. Then, based on the volume of invalid data and the size of each small storage block, the proportion of invalid data in each small storage block can be determined.

[0028] Step 103 : Determine a first garbage collection threshold corresponding to each garbage collection object according to the proportion of invalid data in each small storage block.

[0029] After determining the proportion of invalid data in each small storage block, the first garbage collection threshold corresponding to each garbage collection object can be further determined. Specifically, a proportion threshold can be determined first. Then, the number of small storage blocks whose proportion is greater than the threshold is divided by the number of all small storage blocks, and the resulting ratio is used as the first garbage collection threshold. Alternatively, a quantity threshold can be determined first, starting from the highest proportion, and in descending order, the number of small storage blocks corresponding to each proportion is added up, until the sum exceeds the quantity threshold for the first time. At this time, the corresponding proportion value is used as the first garbage collection threshold.

[0030] Step 104 : Determine a second garbage collection threshold corresponding to each storage unit based on the first garbage collection threshold corresponding to each garbage collection object.

[0031] After determining the first garbage collection threshold corresponding to each garbage collection object, the second garbage collection threshold corresponding to each storage unit can be further determined. Specifically, the first garbage collection thresholds corresponding to each garbage collection object belonging to the same storage unit can be weighted, and the resulting value can be used as the second garbage collection threshold corresponding to the storage unit. The weight here can be pre-set or determined based on the number of small storage blocks included in each garbage collection object.

[0032] Step 105: Perform garbage collection on each small storage block according to the second garbage collection threshold.

[0033] After determining the second garbage collection threshold corresponding to each storage unit, garbage collection can be performed on each small storage block based on the second garbage collection threshold. Specifically, data in each small storage block can be identified to determine valid data and invalid data. If the proportion of invalid data exceeds the second garbage collection threshold, garbage collection can be performed on each small storage block.

[0034] Compared with the existing garbage collection method, the hard disk garbage collection method provided by the above-mentioned embodiment of the present disclosure splits the garbage collection objects into small granularity, and determines the appropriate garbage collection threshold based on statistics, thereby improving the efficiency of garbage collection and effectively reducing the risk of data space being full.

[0035] Continue to see Figure 2 , which shows a process 200 of another embodiment of the hard disk garbage collection method according to the present disclosure. Figure 2 As shown, the method in this embodiment may include the following steps:

[0036] Step 201 : for each redundancy type corresponding to a single garbage collection object in each storage unit in the hard disk, determine the size of the small storage block corresponding to the redundancy type.

[0037] In this embodiment, when dividing individual garbage collection objects in each storage unit in the hard disk, the redundancy type corresponding to each garbage collection object can be considered. Here, the redundancy type may include but is not limited to: replicas, erasure code (EC) redundancy. Replicas refer to redundant storage of data, that is, multiple independent copies of data are stored on different servers. The purpose of this method is to ensure the reliability and availability of data. Even if some servers fail, other copies of the data can still be accessed and used. Erasure code redundancy is a data redundancy technology that generates redundant data blocks (check blocks) by dividing the original data into multiple data blocks and encoding these data blocks to provide data fault tolerance.

[0038] The size of the corresponding small storage block can be determined according to the redundancy type. Specifically, the size of the corresponding small storage block can be determined according to the number of data slices corresponding to the redundancy type. For example, when the redundancy type is N copies, the size of the small storage block can be set to a preset value, or to a power of 2 value closest to N. Taking N=3 as an example, the size of the small storage block can be set to 4M. The erasure code type is calculated as data slice * 1M, such as EC2+1, EC2+2, the data slice is 2, and it is calculated as 2M. EC4+1, EC4+2 has 4 data slices, which is calculated as 4M, EC8+1, EC8+2 and so on are similar.

[0039] Step 202 : For each garbage collection object, the garbage collection object is divided according to the sizes of the small storage blocks corresponding to the redundancy types corresponding to the garbage collection object to obtain a plurality of small storage blocks.

[0040] After determining the size of the small storage blocks corresponding to each redundancy type, each garbage collection object can be partitioned based on the size to obtain multiple small storage blocks. It will be appreciated that different redundancy types correspond to different sizes of small storage blocks. When partitioning garbage collection objects, multiple partitioning schemes can be obtained. The number of partitioning schemes is the same as the number of small storage blocks.

[0041] Step 203: For each small storage block, determine the proportion of invalid data in the small storage block.

[0042] Step 204: For each redundancy type corresponding to a single garbage collection object, determine the number of small storage blocks corresponding to each proportion based on the proportion of invalid data in each small storage block divided according to the redundancy type; and determine the first garbage collection threshold corresponding to the different redundancy types corresponding to the single garbage collection object based on the number of small storage blocks corresponding to each proportion.

[0043] After determining the proportion of garbage data in each small storage block, the number of small storage blocks corresponding to each proportion value can be determined. Specifically, a series of small storage blocks divided according to a single redundancy type can be analyzed. That is, for redundancy type A, the corresponding small storage block size is M. According to the division of M, N small storage blocks are obtained. Based on the proportion of invalid data in the N small storage blocks, the number of small storage blocks corresponding to a single proportion can be counted. Then, based on the number of small storage blocks corresponding to each proportion, the first garbage collection threshold corresponding to the different redundancy types corresponding to a single garbage collection object is determined. Specifically, the proportion corresponding to the highest value can be used as the first garbage collection threshold.

[0044] In some optional implementations of this embodiment, a curve can be drawn with the percentage as the horizontal axis and the number of small storage blocks as the vertical axis. It is understandable that different redundancy types correspond to different curves. Then, based on the curve and a preset threshold, a first garbage collection threshold is determined. The preset threshold can be understood as the area value between the curve, the horizontal axis, and the vertical line corresponding to the first garbage collection threshold, or it can be understood as the ratio of the above area value to the total area value of the area enclosed by the curve and the two coordinate axes.

[0045] Combine Figure 3 , Figure 3The figure shows the relationship between the proportion of invalid data in small storage blocks and the number of small storage blocks. The preset threshold can be set to 1 / 4, which represents the ratio of the area covered by the vertical line to the area between the curve and the two coordinate axes. Based on this ratio, the horizontal coordinate value corresponding to the leftmost vertical line is determined to be the first garbage collection threshold.

[0046] Step 205 : For each redundancy type, weight the first garbage collection thresholds corresponding to the redundancy type to determine the second garbage collection thresholds corresponding to the different redundancy types of each storage unit.

[0047] After determining the first garbage collection threshold corresponding to each redundancy type of a single garbage collection object, further statistics can be performed on each garbage collection object. Specifically, the first garbage collection threshold corresponding to each garbage collection object in a single storage unit can be weighted by redundancy type to determine the second garbage collection threshold corresponding to different redundancy types of the single storage unit.

[0048] When determining the weight of each first garbage collection threshold, it can be determined according to the number of small storage blocks included in each garbage collection object. The more small storage blocks a garbage collection object includes, the greater the weight of the first garbage collection threshold.

[0049] Step 206: For each garbage collection object, in response to determining that the proportion of invalid data in the garbage collection object is greater than the second garbage collection threshold, garbage collection is performed on small storage blocks whose proportion of invalid data in the garbage collection object is greater than the second garbage collection threshold.

[0050] After determining the second garbage collection threshold corresponding to each redundancy type, garbage collection can be performed on each small storage block. Specifically, it can be first determined whether the proportion of invalid data in each garbage collection object is greater than the second garbage collection threshold. It can be understood that if a single garbage collection object corresponds to multiple redundancy types, and the second garbage collection thresholds corresponding to each redundancy type are different, as long as the proportion of invalid data in the single garbage collection object is greater than the smallest value among the second garbage collection thresholds, further judgment can be made on the small storage blocks in the garbage collection object. That is, garbage collection is performed on the small storage blocks in which the proportion of invalid data in the garbage collection object is greater than the second garbage collection threshold. In some specific practices, if the data stored in a single small storage block contains multiple redundancy types, and the second garbage collection thresholds corresponding to each redundancy type are different. When garbage collection is performed on the small storage block, the proportion of invalid data can be compared with the smallest value among the second garbage collection thresholds.

[0051] In some optional implementations of this embodiment, when performing garbage collection on a small storage block, the valid data in the small storage block can be first migrated to other free storage units, and then the invalid data in the small storage block can be deleted to free up space in the small storage block.

[0052] Combine Figure 4 To understand, Figure 4 The figure shows a schematic diagram of the process of determining the second garbage collection threshold in the hard disk garbage collection method disclosed in the present invention. The producer performs garbage scanning on a single storage unit rank of the hard disk to obtain multiple garbage collection objects tgt0~tgtN. For each garbage collection object tgt, after dividing it according to the redundancy type (redundancy 1~redundancy N shown in the figure), multiple small storage blocks are obtained. Determine the proportion of invalid data in each small storage block. Count the number of small storage blocks according to the proportion from 0 to 100, and obtain the proportion corresponding to different redundancy types. According to the recycling threshold determination rule, that is, the preset threshold is 1 / 4, and the first garbage collection threshold TWL corresponding to each redundancy type is determined. Then, each first garbage collection threshold TWL is weighted to obtain the second recycling threshold RWL.

[0053] The hard disk garbage collection method provided by the above-mentioned embodiments of the present disclosure can split a larger garbage collection object into smaller granularity to obtain multiple small storage blocks. The method also calculates the proportion of invalid data in each small storage block to obtain the optimal garbage collection threshold, thereby reducing the amount of valid data moved during the garbage collection process, improving garbage collection efficiency, and effectively reducing the risk of data space being full.

[0054] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a hard disk garbage collection device. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0055] like Figure 5 As shown, the hard disk garbage collection device 500 of this embodiment includes: an object division unit 501 , a proportion determination unit 502 , a first threshold determination unit 503 , a second threshold determination unit 504 and a garbage collection unit 505 .

[0056] The object partitioning unit 501 is configured to partition each garbage collection object in each storage unit in the hard disk to obtain a plurality of small storage blocks.

[0057] The proportion determining unit 502 is configured to determine, for each small storage block, a proportion of invalid data in the small storage block.

[0058] The first threshold determination unit 503 is configured to determine a first garbage collection threshold corresponding to each garbage collection object according to a proportion of invalid data in each small storage block.

[0059] The second threshold determination unit 504 is configured to determine a second garbage collection threshold corresponding to each storage unit according to the first garbage collection threshold corresponding to each garbage collection object.

[0060] The garbage collection unit 505 is configured to perform garbage collection on each small storage block according to a second garbage collection threshold.

[0061] In addition, in the technical solution of this application, an electronic device is also proposed.

[0062] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0063] like Figure 6 As shown, the electronic device may include a processor 601, a memory 602, a bus 603, and a computer program stored in the memory 602 and executable on the processor 601, wherein the processor 601 and the memory 602 communicate with each other via the bus 603. When the processor 601 executes the computer program, the steps of the above method are implemented, for example, including: for each storage unit in the hard disk, dividing each garbage collection object in the storage unit to obtain multiple small storage blocks; for each small storage block, determining the proportion of invalid data in the small storage block; determining a first garbage collection threshold corresponding to each garbage collection object based on the proportion of invalid data in each small storage block; determining a second garbage collection threshold corresponding to each storage unit based on the first garbage collection threshold corresponding to each garbage collection object; and performing garbage collection on each small storage block based on the second garbage collection threshold.

[0064] In addition, in one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented, for example, including: for each storage unit in the hard disk, each garbage collection object in the storage unit is divided to obtain multiple small storage blocks; for each small storage block, the proportion of invalid data in the small storage block is determined; based on the proportion of invalid data in each small storage block, the first garbage collection threshold corresponding to each garbage collection object is determined; based on the first garbage collection threshold corresponding to each garbage collection object, the second garbage collection threshold corresponding to each storage unit is determined; and based on the second garbage collection threshold, each small storage block is garbage collected.

[0065] In summary, the technical solution disclosed herein can partition each garbage collection pair within each storage unit on a hard drive to obtain multiple small storage blocks. The percentage of garbage data in each small storage block is then calculated. Based on this percentage, a first garbage collection threshold corresponding to each redundancy type is determined. Furthermore, a second garbage collection threshold corresponding to each redundancy type in each storage unit is determined. Finally, garbage collection is performed on each small storage block based on the second garbage collection threshold. This reduces the amount of data moved while improving recycling efficiency.

[0066] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A hard disk garbage collection method, comprising: For each storage unit in the hard disk, each garbage collection object in the storage unit is divided to obtain multiple small storage blocks, wherein each garbage collection object corresponds to at least one redundancy type; For each small storage block, determine the proportion of invalid data in the small storage block; Determine a first garbage collection threshold corresponding to each garbage collection object based on a proportion of invalid data in each small storage block; Determining a second garbage collection threshold corresponding to each storage unit according to the first garbage collection threshold corresponding to each garbage collection object; performing garbage collection on each small storage block according to the second garbage collection threshold; The garbage collection objects in the storage unit are divided into multiple small storage blocks, including: For each redundancy type corresponding to a single garbage collection object, determine the size of the small storage block corresponding to the redundancy type; For each garbage collection object, the garbage collection object is divided according to the size of the small storage blocks corresponding to each redundancy type corresponding to the garbage collection object to obtain multiple small storage blocks; Determining the first garbage collection threshold corresponding to each garbage collection object according to the proportion of invalid data in each small storage block includes: For each redundancy type corresponding to a single garbage collection object, determine the number of small storage blocks corresponding to each proportion based on the proportion of garbage data in each small storage block divided according to the redundancy type; Determining first garbage collection thresholds corresponding to different redundancy types corresponding to a single garbage collection object based on the number of small storage blocks corresponding to each proportion; The determining, based on the number of small storage blocks corresponding to each proportion, first garbage collection thresholds corresponding to different redundancy types corresponding to a single garbage collection object includes: For each redundancy type, plot the curve corresponding to the redundancy type, with the percentage of garbage data in each small storage block divided according to the redundancy type as the horizontal axis and the number of small storage blocks corresponding to the percentage as the vertical axis; According to the curve and the preset threshold, a first garbage collection threshold corresponding to different redundancy types of a single garbage collection object is determined.

2. The method according to claim 1, wherein The determining, based on the first garbage collection threshold corresponding to each garbage collection object, the second garbage collection threshold corresponding to each storage unit includes: For each redundancy type, weights are applied to the first garbage collection thresholds corresponding to the redundancy type to determine the second garbage collection thresholds corresponding to the different redundancy types of each storage unit.

3. The method according to claim 1, wherein The performing garbage collection on each small storage block according to the second garbage collection threshold includes: For each garbage collection object, in response to determining that the proportion of invalid data in the garbage collection object is greater than the second garbage collection threshold, garbage collection is performed on small storage blocks in which the proportion of invalid data in the garbage collection object is greater than the second garbage collection threshold.

4. The method according to claim 3, wherein: The performing garbage collection on the small storage block where the proportion of invalid data in the garbage collection object is greater than the second garbage collection threshold includes: The valid data in the small storage blocks in which the proportion of invalid data in the garbage collection object is greater than the second garbage collection threshold is migrated, and the invalid data is deleted.

5. A hard disk garbage collection device, comprising: The object partitioning unit is configured to partition each garbage collection object in each storage unit in the hard disk to obtain a plurality of small storage blocks, wherein each garbage collection object corresponds to at least one redundancy type; a proportion determining unit configured to determine, for each small storage block, a proportion of invalid data in the small storage block; A first threshold determination unit is configured to determine a first garbage collection threshold corresponding to each garbage collection object according to a proportion of invalid data in each small storage block; a second threshold determination unit configured to determine a second garbage collection threshold corresponding to each storage unit according to the first garbage collection threshold corresponding to each garbage collection object; a garbage collection unit configured to perform garbage collection on each small storage block according to the second garbage collection threshold; The object partitioning unit is further configured to: For each redundancy type corresponding to a single garbage collection object, determine the size of the small storage block corresponding to the redundancy type; For each garbage collection object, the garbage collection object is divided according to the size of the small storage blocks corresponding to each redundancy type corresponding to the garbage collection object to obtain multiple small storage blocks; The first threshold determination unit is further configured to: For each redundancy type corresponding to a single garbage collection object, determine the number of small storage blocks corresponding to each proportion based on the proportion of garbage data in each small storage block divided according to the redundancy type; For each redundancy type, plot the curve corresponding to the redundancy type, with the percentage of garbage data in each small storage block divided according to the redundancy type as the horizontal axis and the number of small storage blocks corresponding to the percentage as the vertical axis; According to the curve and the preset threshold, a first garbage collection threshold corresponding to different redundancy types of a single garbage collection object is determined.

6. An electronic device comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the hard disk garbage collection method according to any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the hard disk garbage collection method according to any one of claims 1 to 4 is implemented.

Images