A method and device for reducing data migration during garbage data recovery

Abstract

The present invention discloses a method and device for reducing data migration during garbage data collection, and relates to the field of computer storage technology. The method for reducing data migration during garbage data collection includes the following steps: designing a variable, namely MoveCount, which is used to save the number of migrations, then judging the MoveCount of each data block, and performing data migration according to the result of MoveCount judgment. The present invention mainly proposes an optimization scheme for reducing data migration in a garbage collection scenario. During garbage collection, the hotness and coldness of data are identified by the number of migrations, and then the data is placed in different storage spaces according to the hotness and coldness. The advantage is that the amount of data migration can be reduced, the data migration process can be optimized, garbage collection can be completed quickly, the impact on the client's read performance can be reduced, and the consumption of CPU resources can be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of computer storage, and in particular to a method and device for reducing data migration during garbage data recovery. Background Art

[0002] With the continuous development of cloud computing and the explosive growth of data volume, new requirements are put forward for storage systems. Storage systems are required to be deployed on general-purpose, cheap servers, provide horizontal expansion capabilities, and provide extreme performance. As a result, a "write-append" type of storage system has emerged. The principle is as follows: Figure 6 shown.

[0003] The upper-layer application performs random overwrite writes on BLOCKs, which are converted by the system into sequential append writes on CHUNKs. In this way, the disk locality principle can be fully utilized to greatly improve the write performance. In order to achieve this conversion, address redirection exists to save the address mapping relationship between the overwrite write area and the append write area, such as Figure 7 shown.

[0004] from Figure 6 It can be seen that when overwriting a BLOCK, the data written first will be overwritten by the data written later. However, since the system has converted it into an append write mode for a CHUNK, data overwriting will not occur immediately. Instead, the index is updated to redirect the data to the new CHUNK location and the old data area is marked as invalid. Figure 8 shown.

[0005] As time goes by, or data is constantly overwritten, invalid data will increase, wasting the system's storage space, so the system needs to perform garbage collection regularly. The commonly used garbage collection method in the industry is: when the invalid data in a CHUNK reaches a certain proportion, the valid data in it can be migrated to a new CHUNK, and the storage space occupied by the old CHUNK can be released. Fig. 9 As shown, the valid data in CHUNK1 and CHUNK2 are migrated to the new CHUNK3. After the migration is completed, the storage space of CHUNK1 and CHUNK2 can be released.

[0006] However, when garbage collection is performed, the migration of valid data will have a certain impact on the upper-layer data reading performance. At the same time, frequent data migration will also cause a certain consumption of CPU resources. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides a method and device for reducing data migration during garbage data recovery, which solves the problem that when performing garbage data recovery, the migration of valid data will have a certain impact on the upper layer's data reading performance, and frequent data migration will also cause a certain consumption of CPU resources.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions: A method for reducing data migration during garbage data recovery, comprising the following method steps:

[0009] S1: Design a variable, MoveCount, which is used to save the number of migrations;

[0010] S2: Determine the MoveCount of each data block. The probability of a CHUNK distributed in the cold data space being overwritten is very small, so the proportion of invalid data is small, and the probability of valid data being migrated is small; the probability of a CHUNK distributed in the hot data space is very high, so the proportion of invalid data is large, and when performing garbage collection, only a small amount of valid data needs to be migrated;

[0011] S3: Execute data migration according to the MoveCount judgment result to reduce the amount of data migration and quickly complete garbage collection.

[0012] Preferably, in S1, the heat of data is determined by the number of migrations. If a data block is migrated more times, it means that the heat of data around the data block is higher. Since the surrounding data is often overwritten, a large amount of invalid data is generated early. Therefore, the data block is frequently migrated. The greater the number of migrations, the data block is defined as cold data compared with the surrounding hot data. Therefore, it can be defined that the greater the number of migrations, the colder the surface data.

[0013] Preferably, in S2, the MoveCount of the data block is determined based on the following:

[0014] If MoveCount<3, it falls into the hot data space;

[0015] If 3<=MoveCount<10, it falls into the warm data space;

[0016] If MoveCount>=10, it falls into the cold data space.

[0017] Preferably, in S3, the specific steps of data migration are as follows:

[0018] S31, dividing the CHUNK into a plurality of data units according to a fixed size, and recording a MoveCount for each data unit, which is stored in the metadata area of ​​the CHUNK;

[0019] S32. If the upper layer application writes data to the CHUNK, MoveCount is assigned a value of 0; if data from other CHUNKs is migrated to the new CHUNK, MoveCount is incremented by 1.

[0020] S33, start garbage collection;

[0021] S34, read the CHUNK list and allocate BIT bits of memory space for each CHUNK;

[0022] S35, scan the BLOCK->CHUNK mapping table, and update the status of the BIT memory space according to the mapping situation, with valid data being 1 and invalid data being 0; and record the valid space ratio of the CHUNK;

[0023] S36. According to the recycling setting strategy, determine the CHUNK that needs to be garbage collected;

[0024] S37, read the data in the CHUNK to be recycled, check the migration count information in the meta information, and write the data to different CHUNK spaces according to the MoveCount count;

[0025] S38. After all data in the CHUNK is migrated, the metadata information is updated and the data is downloaded to the disk. After the update is successful, the old CHUNK is deleted to release storage space.

[0026] S39. Update data CHUNK list information.

[0027] Preferably, in S31, when writing data, metadata is written to the disk together with the CHUNK.

[0028] Preferably, in S33, a recycling time strategy can be customized, such as starting once a day when idle.

[0029] Preferably, in said S34, the BIT memory space is used to record the data status of each data unit, including valid data and invalid data.

[0030] Preferably, in S36, the recycling strategy is as follows: recycling is performed when the CHUNK garbage space reaches 50%, which is unconditional recycling; recycling is performed when the CHUNK garbage space reaches 20%, which is recycling when the business is idle; recycling is performed when the CHUNK garbage space reaches 10%, which is deep recycling.

[0031] Preferably, in S37, before each writing, the MoveCount recorded in the meta needs to be increased by 1.

[0032] The present invention also provides a device for reducing data migration during garbage data recovery, comprising a variable design module, a variable judgment module and a data migration module, wherein:

[0033] The variable design module is used to design a variable, namely MoveCount, which is used to save the number of migrations;

[0034] Variable judgment module, used to judge the MoveCount of each data block;

[0035] The data migration module is used to perform data migration according to the MoveCount judgment result, thereby reducing the amount of data migration and quickly completing garbage collection.

[0036] The present invention has the following beneficial effects:

[0037] (1) The method for reducing data migration during garbage collection mainly proposes an optimization scheme for reducing data migration in the garbage collection scenario. During garbage collection, the hotness of the data is identified by the number of migrations, and then the data is placed in different storage spaces according to the hotness. The advantage is that the amount of data migration can be reduced, the data migration process can be optimized, garbage collection can be completed quickly, the impact on the client read performance can be reduced, and the consumption of CPU resources can be reduced.

[0038] (2) A method for reducing data migration during garbage collection is to design a MoveCount. After determining the MoveCount of each data block, it can be seen that the probability of a CHUNK distributed in a cold data space being overwritten is very small, so the proportion of invalid data therein is small, and the probability of valid data being migrated becomes smaller; the probability of a CHUNK distributed in a hot data space being overwritten is very high, so the proportion of invalid data therein is large. Therefore, when performing garbage collection, only a small amount of valid data needs to be migrated, thereby obtaining a data processing standard that reduces the amount of data migration.

[0039] (3) A method for reducing data migration during garbage data recovery is to first update the metadata information and save it to disk after all data in the CHUNK is migrated. After the update is successful, the old CHUNK is deleted to release storage space. This allows most of the data stored in one CHUNK to be valid data, while most of the data stored in another CHUNK to be invalid data, thereby achieving the goal of reducing the amount of data migration.

[0040] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the method of the present invention;

[0042] Figure 2 The data migration flow chart of the present invention;

[0043] Figure 3 This is a data judgment flow chart of the present invention;

[0044] Figure 4 It is the initial data state record diagram of the present invention;

[0045] Figure 5 It is a data state record diagram of the intermediate process of the present invention;

[0046] Figure 6 It is a diagram showing the existing storage system of the present invention;

[0047] Figure 7 is the existing mapping relationship table of the present invention;

[0048] Figure 8 It is a schematic diagram of existing data marking of the present invention;

[0049] Fig. 9 It is a schematic diagram of existing data migration of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] In the description of the present invention, it should be understood that the terms "opening", "upper", "lower", "thickness", "top", "middle", "length", "inside", "all around" and the like indicating orientation or positional relationship are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the components or elements referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0052] See also Figure 1-Figure 5, the embodiment of the present invention provides a technical solution: when valid data and invalid data are intertwined, and the invalid data reaches a certain proportion, the garbage collection mechanism will be triggered, and the valid data will be migrated once to a new CHUNK. The present invention designs a mechanism to make most of the data stored in a part of the CHUNK valid data; and most of the data stored in another part of the CHUNK is invalid data, so the goal of reducing the amount of data migration can be achieved. For example, the same 4 valid data blocks need to be migrated 4 times before optimization, but only need to be migrated once after the data distribution is changed through optimization;

[0053] Therefore, a method for reducing data migration during garbage data collection is proposed, including the following method steps: Figure 1 As shown:

[0054] S1: Design a variable, MoveCount, which is used to save the number of migrations;

[0055] S2: Determine the MoveCount of each data block. The probability of a CHUNK distributed in the cold data space being overwritten is very small, so the proportion of invalid data is small, and the probability of valid data being migrated is small; the probability of a CHUNK distributed in the hot data space is very high, so the proportion of invalid data is large, and when performing garbage collection, only a small amount of valid data needs to be migrated;

[0056] S3: Execute data migration according to the MoveCount judgment result to reduce the amount of data migration and quickly complete garbage collection.

[0057] Furthermore, in S1, the heat of data is determined by the number of migrations. If a data block is migrated more times, it means that the data around the data block is hotter. Since the surrounding data is often overwritten, a large amount of invalid data is generated. Therefore, the data block is frequently migrated, and the greater the number of migrations, the data block is defined as cold data compared with the surrounding hot data. Therefore, it can be defined that the greater the number of migrations, the colder the surface data.

[0058] Furthermore, in S2, the MoveCount of the data block is determined based on the following: Figure 3 As shown:

[0059] If MoveCount<3, it falls into the hot data space;

[0060] If 3<=MoveCount<10, it falls into the warm data space;

[0061] If MoveCount>=10, it falls into the cold data space.

[0062] Furthermore, in S3, the specific steps of data migration are as follows: Figure 2 As shown:

[0063] S31. Divide the CHUNK into many data units (SLICE) according to a fixed size (such as 4k). Each data unit records a MoveCount, which is stored in the metadata area (meta) of the CHUNK. When writing data, the metadata will be written to the disk together with the CHUNK;

[0064] S32. If the upper layer application writes data to the CHUNK, MoveCount is assigned a value of 0; if data from other CHUNKs is migrated to the new CHUNK, MoveCount is incremented by 1.

[0065] S33, start garbage collection, you can customize the collection time strategy, such as starting it once a day when idle;

[0066] S34, read the CHUNK list, and allocate a BIT bit memory space for each CHUNK, the BIT bit memory space is used to record the data status of each data unit, including valid data and invalid data, such as Figure 4 As shown, at this time, the effective space of CHUNK is in the initialization state;

[0067] S35, scan the BLOCK->CHUNK mapping table, and update the status of the BIT memory space according to the mapping situation, with valid data being 1 and invalid data being 0; and record the effective space ratio of the CHUNK, such as Figure 5 As shown, at this time, the effective space of CHUNK is less than 50%;

[0068] S36. According to the recycling setting strategy, determine the CHUNK that needs to be garbage collected;

[0069] S37, read the data in the CHUNK to be recycled, check the migration count information in the meta information, and write the data to different CHUNK spaces according to the MoveCount count. Before each write, the MoveCount recorded in the meta needs to be increased by 1;

[0070] S38. After all data in the CHUNK is migrated, the metadata information is updated and the data is downloaded to the disk. After the update is successful, the old CHUNK is deleted to release storage space. This allows most of the data stored in some CHUNKs to be valid data, while most of the data stored in other CHUNKs to be invalid data, thereby reducing the amount of data migration.

[0071] S39. Update data CHUNK list information.

[0072] Furthermore, in S36, the recycling strategy is as follows: when the CHUNK garbage space reaches 50%, it is recycled, which is unconditional recycling; when the CHUNK garbage space reaches 20%, it is recycled when the business is idle; when the CHUNK garbage space reaches 10%, it is recycled, which is deep recycling.

[0073] The present invention also provides a device for reducing data migration during garbage data recovery, comprising a variable design module, a variable judgment module and a data migration module, wherein:

[0074] The variable design module is used to design a variable, namely MoveCount, which is used to save the number of migrations;

[0075] Variable judgment module, used to judge the MoveCount of each data block;

[0076] The data migration module is used to perform data migration according to the MoveCount judgment result, thereby reducing the amount of data migration and quickly completing garbage collection.

[0077] The present invention mainly proposes an optimization scheme to reduce data migration in the garbage collection scenario. During garbage collection, the hotness of the data is identified by the number of migrations, and then placed in different storage spaces according to the hotness and coldness. The advantage is that the amount of data migration can be reduced, the data migration process can be optimized, garbage collection can be completed quickly, the impact on client read performance can be reduced, and the consumption of CPU resources can be reduced.

[0078] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0079] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for reducing data migration during garbage data collection, characterized in that: The method comprises the following steps: S1: Design a variable, MoveCount, which is used to save the number of migrations; S2: Determine the MoveCount of each data block; S3: Execute data migration according to the MoveCount judgment result to reduce the amount of data migration and quickly complete garbage collection; The specific steps of data migration in S3 are as follows: S31, dividing the CHUNK into a plurality of data units according to a fixed size, and recording a MoveCount for each data unit, which is stored in the metadata area of ​​the CHUNK; S32. If the upper layer application writes data to the CHUNK, MoveCount is assigned a value of 0; if data from other CHUNKs is migrated to the new CHUNK, MoveCount is incremented by 1. S33, start garbage collection; S34, read the CHUNK list and allocate BIT bits of memory space for each CHUNK; S35, scan the BLOCK->CHUNK mapping table, and update the status of the BIT memory space according to the mapping situation, with valid data being 1 and invalid data being 0; and record the valid space ratio of the CHUNK; S36. According to the recycling setting strategy, determine the CHUNK that needs to be garbage collected; S37, read the data in the CHUNK to be recycled, check the migration count information in the meta information, and write the data to different CHUNK spaces according to the MoveCount count; S38. After all data in the CHUNK is migrated, the metadata information is updated and the data is downloaded to the disk. After the update is successful, the old CHUNK is deleted to release storage space. S39, update data CHUNK list information; Wherein, in said S31, when writing data, metadata will be written to the disk together with CHUNK.

2. The method for reducing data migration during garbage data recovery according to claim 1, characterized in that: In S1, the heat of the data is determined by the number of migrations, and it can be defined that the greater the number of migrations, the colder the surface data.

3. The method for reducing data migration during garbage data recovery according to claim 1, characterized in that: In S2, the MoveCount of the data block is determined based on the following: If MoveCount<3, it falls into the hot data space; If 3<=MoveCount<10, it falls into the warm data space; If MoveCount>=10, it falls into the cold data space.

4. The method for reducing data migration during garbage data recovery according to claim 1, characterized in that: In S33, the recycling time strategy can be customized, including starting once a day when idle.

5. The method for reducing data migration during garbage data recovery according to claim 1, characterized in that: In the above S34, the BIT memory space is used to record the data status of each data unit, including valid data and invalid data.

6. The method for reducing data migration during garbage data recovery according to claim 1, characterized in that: In S36, the recycling strategy is specifically as follows: recycling is performed when the CHUNK garbage space reaches 50%, which is unconditional recycling; recycling is performed when the CHUNK garbage space reaches 20%, which is recycling when the business is idle; recycling is performed when the CHUNK garbage space reaches 10%, which is deep recycling.

7. The method for reducing data migration during garbage data recovery according to claim 1, characterized in that: In S37, before each writing, the MoveCount recorded in the meta needs to be increased by 1.

8. A device for reducing data migration during garbage data recovery, characterized in that: It includes variable design module, variable judgment module and data migration module, among which: The variable design module is used to design a variable, namely MoveCount, which is used to save the number of migrations; Variable judgment module, used to judge the MoveCount of each data block; The data migration module is used to perform data migration according to the MoveCount judgment result, thereby reducing the amount of data migration and quickly completing garbage collection; The specific steps of data migration are as follows: S31, dividing the CHUNK into a plurality of data units according to a fixed size, and recording a MoveCount for each data unit, which is stored in the metadata area of ​​the CHUNK; S32. If the upper layer application writes data to the CHUNK, MoveCount is assigned a value of 0; if data from other CHUNKs is migrated to the new CHUNK, MoveCount is incremented by 1. S33, start garbage collection; S34, read the CHUNK list and allocate BIT bits of memory space for each CHUNK; S35, scan the BLOCK->CHUNK mapping table, and update the status of the BIT memory space according to the mapping situation, with valid data being 1 and invalid data being 0; and record the valid space ratio of the CHUNK; S36. According to the recycling setting strategy, determine the CHUNK that needs to be garbage collected; S37, read the data in the CHUNK to be recycled, check the migration count information in the meta information, and write the data to different CHUNK spaces according to the MoveCount count; S38. After all data in the CHUNK is migrated, the metadata information is updated and the data is downloaded to the disk. After the update is successful, the old CHUNK is deleted to release storage space. S39, update data CHUNK list information; Wherein, in said S31, when writing data, metadata will be written to the disk together with CHUNK.

Images