Method, device and equipment for dynamic grouping and garbage collection based on data updating characteristics

Abstract

The invention provides a method, a device and equipment for dynamic grouping and garbage collection based on data updating characteristics, which are used for solving the technical problem that the write-in amplification factor WAF of a storage system is relatively large. According to the method, the logic data blocks are dynamically grouped based on the updating frequency and the periodic characteristics of the logic data blocks, and the number of segments matched with the characteristics of the groups and the garbage collection strategy are allocated for different types of groups, so that the overall write-in amplification factor WAF of the storage system is optimized, and the purposes of prolonging the service life of a hard disk, improving the write-in performance and improving the storage efficiency are achieved by optimizing the WAF. And the system stability is improved.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a method, device and equipment for dynamic grouping and garbage collection based on data update characteristics. Background Art

[0002] Log-Structured Systems is a design method for data management and storage systems. Its main feature is to record all modification operations in the form of appended logs. In this system, all data write operations are sequentially appended to the end of the log file instead of directly modifying the original data location on the disk. This design method can greatly optimize performance in scenarios with frequent writes.

[0003] Log-Structured Systems are widely used in various scenarios, with typical applications such as key-value KV storage (such as LSM tree) systems, file storage systems (such as F2FS), storage firmware (such as Flash Translation Layer, FTL) systems, etc. Log-Structured Systems can greatly improve write throughput and are very suitable for emerging storage media that support append writes, such as NAND SSD disks, ZNS disks, SMR HDD disks and other types of hard disks.

[0004] Despite these benefits, the high garbage collection (GC) cost of log-structured systems is their main drawback. Log-structured systems divide storage space into fixed-size segments, each of which is usually several MB in size, and the segment itself contains several 4KB data blocks. Segments are the unit of space allocation and GC, and 4KB blocks are the unit of reading and writing data. Log-structured systems append updated data blocks to the current segment, leaving old versions of data blocks as garbage, which must be cleaned and recycled later by GC.

[0005] During garbage collection (GC), the victim segment with garbage blocks to be recycled is first found, then the valid blocks are copied to another segment, and finally the entire victim segment is released for future writing. It is not difficult to see that GC will cause a lot of additional read and write operations.

[0006] Write Amplification Factor (WAF) is a common indicator for measuring the impact of additional writes during GC. Write amplification in a storage system refers to the phenomenon that the amount of data actually written to a storage device is greater than the amount of data the user expects to write during a write operation. Write amplification may lead to a decrease in storage system performance, shorten the life of storage devices, and reduce the stability of the storage system.

[0007] The write amplification factor (WAF) is the ratio of the total number of blocks written to storage to the number of blocks written by users:

[0008]

[0009] At present, the research on reducing the overhead caused by GC mainly focuses on two aspects: how to classify and place data according to its life cycle (putting data with the same or similar life cycle into the same segment), and how to select the victim segment (the selected segment will be GC). Of these two problems, the first one is the key; the solution to the second one is closely related to the first one.

[0010] Ideally, if data blocks with the same life cycle are placed in a segment, the data in this segment will become invalid almost at the same time. In this case, no additional data copy is required, and WAF reaches the optimal value of 1.

[0011] In actual all-flash storage systems, for the convenience of management and implementation, the use space of SSD as the storage medium is often further grouped. A common method is to group a number of consecutive fixed data blocks into a data segment (called a segment or segment). In this way, an SSD disk will contain several segments. In order to give full play to the performance of the storage system, the data writing rules meet the following requirements:

[0012] The storage system is implemented based on log-write.

[0013] The storage system can write to multiple segments concurrently.

[0014] For a particular segment, data is written sequentially, and when it is full, the remaining free segments are written.

[0015] Garbage collection is also performed in segments.

[0016] Based on the above requirements, we can know that at the beginning, the system is full of free segments. As data is written, the free segments gradually decrease. At the same time, data updates will cause the garbage data in the segments to gradually increase. At a certain point, the system needs to perform garbage collection to obtain free segments. At this time, in order to obtain the minimum write amplification, two questions must be answered:

[0017] 1) How is the user's written data laid out in the segment (because data can be written to multiple data segments concurrently)?

[0018] 2) What strategy should be adopted when performing garbage collection?

[0019] The data life cycle refers to the time that data exists before it is updated. After the data is updated, the previously written data becomes garbage data, so if a piece of data is updated multiple times, all the previous data in the system except the last written data becomes garbage data.

[0020] Regarding the question of how to lay out the data written by the user in the segment, if we look at it from a higher perspective, when writing data, data with the same life cycle or a life cycle close to the same life cycle should be put into the same segment. In this way, the data of the entire segment will become garbage data in almost the same time. At this time, there is no need to move any valid data to recycle this segment, and the entire segment can be directly erased, and the write amplification is optimized. In actual situations, the system cannot make an accurate judgment on the life cycle of data, so it can only estimate its future life cycle based on the frequency of updates of a data in the past. For example, if a piece of data is updated frequently, it can be considered that there is a high probability that the data will be updated in the future, and this data can be called hot data; relatively speaking, if a piece of data is not updated for a long time, this data can be called cold data. It should be noted that in addition to considering the average number of data updates in the past period of time, the recency information of data updates also needs to be considered when predicting the life cycle of data. For example, data A and B have both been updated 10 times in the past 3 minutes, but data A has been updated in the first minute, while data B has been updated in the third minute. At this time, data A and B have the same update frequency, but data B has a higher recency, which means that data B is more likely to continue to be updated in the short term, that is, data B is more popular. Therefore, when writing data, it is necessary to separate hot and cold data and put them in different segments so that the data in the segments become invalid at approximately the same time to reduce write amplification. To achieve this goal, identifying hot and cold data is key.

[0021] In the prior art, written data blocks are usually simply classified, that is, hot blocks and cold blocks are distinguished. Hot blocks (blocks that are frequently updated) have a shorter lifespan, while cold blocks (blocks that are not frequently updated) have a longer lifespan. Therefore, the heat of IO can be judged based on previous statistical information on the frequency of IO update cycles (it can also be weighted based on the frequency and update time). Based on the pre-set hot and cold thresholds, hot IO is written sequentially to the segment of the hot zone in a log manner; cold IO is written sequentially to the segment of the cold zone in a log manner. Although the concepts of hot and cold are widely accepted, the boundary between hot and cold is related to the workload and usually changes over time. The existing scheme can distinguish hot data more effectively. When the hot data group contains enough segments, WAF can be close to the ideal value of 1.

[0022] However, the existing technical solutions are too rough in classifying the hot and cold data. For the data blocks classified into the cold zone, because they may have different update cycles, they are mixed in the same segment. During the GC, more valid blocks need to be copied from the victim segment, resulting in a larger WAF value. Summary of the invention

[0023] In view of this, the present invention provides a method, apparatus and device for dynamic grouping and garbage collection based on data update characteristics, which are used to solve the technical problem of large write amplification factor WAF of storage system.

[0024] Based on one aspect of an embodiment of the present invention, the present invention provides a method for dynamic grouping and garbage collection based on data update characteristics, the method is applied to a storage system using a log structure system, the method comprising:

[0025] A data update information table reflecting the update frequency characteristics of the logical data block is updated according to the address of the logical data block written by the IO write request; the data update information table records the latest n update interval values ​​of each logical data block, and the update interval value is the difference between the global IO counter value between the current IO write request and the previous IO write request of the logical data block;

[0026] The update interval values ​​of the latest n times of each logical data block in the data update information table are averaged to obtain the update interval mean value, each logical data block is dynamically grouped according to the preset update interval mean value threshold and the periodic characteristics of the n update interval values, and the data grouping of each logical data block is recorded;

[0027] Dynamically adjust the number of segments allocated to each data group according to the percentage of the sum of the update interval averages of all logical data blocks in the data group to the sum of the update interval averages of all logical data blocks;

[0028] Different garbage collection strategies are configured for segments within different data groups.

[0029] Furthermore, in the data update information table, the update interval values ​​of the latest n times are recorded in a circular manner through n fields; and the position of the update interval value that the current IO write request should write is recorded through a position field;

[0030] When calculating the update interval mean value of each logical data block, a weighted average method is used to calculate the update interval mean value, and the update interval value corresponding to the IO write request with a closer occurrence time has a higher weight.

[0031] Further, whether the IO write request for each logical data block has a periodic feature is determined based on the n update interval values ​​recorded in the data update information table by using a preset periodic feature recognition algorithm;

[0032] Based on the update interval mean threshold, the frequently updated logical data blocks are divided into the frequently updated group (C1 group);

[0033] Based on the update interval mean threshold, logical data blocks with update interval mean values ​​within the same threshold range are divided into periodic infrequent update groups (C2 groups);

[0034] All logical data blocks that do not belong to the frequently updated group (C1 group) or the periodically infrequently updated group (C2 group) are divided into the random infrequently updated group (C3 group).

[0035] Furthermore, based on a plurality of update interval mean value thresholds, the logical data blocks in the periodic infrequent update group (C2 group) are further divided into a plurality of sub-data groups.

[0036] Furthermore, for the frequently updated group (C1 group) and the random infrequently updated group (C3 group), a greedy algorithm is configured to perform garbage collection on the data blocks in the segments contained in the data group;

[0037] For data blocks in segments contained in the periodic infrequent update group (C2 group), a benefit-cost algorithm is configured to perform garbage collection.

[0038] Based on one aspect of an embodiment of the present invention, the present invention further provides a device for dynamic grouping and garbage collection based on data update characteristics, the device is applied to a storage system using a log structure system, and the device includes:

[0039] An information table updating module is used to update a data update information table reflecting the update frequency characteristics of the logical data block according to the address of the logical data block written by the IO write request; the data update information table records the latest n update interval values ​​of each logical data block, and the update interval value is the difference between the global IO counter value between the current IO write request and the previous IO write request of the logical data block;

[0040] A dynamic grouping module is used to average the latest n update interval values ​​of each logical data block in the data update information table to obtain an update interval mean value, dynamically group each logical data block according to a preset update interval mean value threshold and the periodic characteristics of the n update interval values, and record the data grouping of each logical data block;

[0041] A dynamic segment allocation module, used to dynamically adjust the number of segments allocated to each data group according to the percentage of the sum of the update interval averages of all logical data blocks in the data group to the sum of the update interval averages of all logical data blocks;

[0042] The recycling strategy setting module is used to configure different garbage collection strategies for segments in different data groups.

[0043] Furthermore, the information table updating module records the latest n update interval values ​​in the data update information table through n fields in a circular manner; and also records the position of the update interval value to which the current IO write request should be written through a position field;

[0044] When calculating the update interval mean value of each logical data block, the information table update module calculates the update interval mean value in a weighted average manner, and the update interval value corresponding to the IO write request with a closer occurrence time has a higher weight.

[0045] Furthermore, the dynamic grouping module includes:

[0046] A periodic feature judgment submodule, used to judge whether the IO write request for each logical data block has a periodic feature based on the n update interval value recorded in the data update information table by using a preset periodic feature recognition algorithm;

[0047] The data grouping submodule is used to divide the frequently updated logical data blocks into the frequently updated group (C1 group) based on the update interval mean threshold; divide the logical data blocks with the update interval mean within the same threshold range into the periodic non-frequent update group (C2 group) based on the update interval mean threshold; and divide all logical data blocks that do not belong to the frequently updated group (C1 group) or the periodic non-frequent update group (C2 group) into the random non-frequent update group (C3 group).

[0048] Furthermore, the recycling strategy setting module configures a greedy algorithm to perform garbage collection on data blocks in segments contained in the frequently updated group (C1 group) and the randomly infrequently updated group (C3 group); and configures a benefit-cost algorithm to perform garbage collection on data blocks in segments contained in the periodically infrequently updated group (C2 group).

[0049] The device provided by the present invention can be implemented in software, hardware or a combination of software and hardware. When implemented in a software module, the program code of the software module is loaded into the storage medium of the device, and the processor reads the program code in the storage medium and executes it. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments of the present invention or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present invention.

[0051] Figure 1 A schematic flow chart of the steps of a method for dynamic grouping and garbage collection based on data update characteristics provided by an embodiment of the present invention;

[0052] Figure 2 A coordinate diagram of the relationship between the data block groups and the update interval mean values ​​of the data block groups in one embodiment of the present invention;

[0053] Figure 3 A schematic diagram of the structure of a device for dynamic grouping and garbage collection based on data update characteristics provided by an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of the structure of an electronic device for implementing the method of dynamic grouping and garbage collection based on data update characteristics according to an embodiment of the present invention is provided. DETAILED DESCRIPTION

[0055] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, rather than limiting the embodiments of the present invention. The singular forms of "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings. Although the embodiments of the present invention may be described in a first, second, or third manner, this description is only used to distinguish similar information, entities, or steps, rather than to describe a specific order or sequence. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. For another example, in some scenarios, the first information may refer to one information or to multiple information of the same type. In addition, the word "if" used may be interpreted as "at..." or "when..." or "in response to determination". The "and / or" in the present invention is only a kind of association relationship describing the associated objects, indicating that there may be three relationships, for example, A and / or B, which may represent: A exists alone, A and B exist at the same time, and B exists alone, wherein A and B may be singular or plural. In addition, in the description of the present invention, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0056] The purpose of the present invention is to provide a technical solution for dynamic grouping and garbage collection based on data update characteristics to reduce the overall write amplification factor WAF of the storage system. The basic idea of ​​the present invention is to dynamically group logical data blocks based on the update frequency and periodic characteristics of logical data blocks, and allocate the number of segments and garbage collection strategies that match the grouping characteristics to different types of groups, thereby optimizing the overall write amplification factor WAF of the storage system, and by optimizing WAF, the technical effects of increasing the service life of the hard disk, improving the write performance, and improving the system stability can be achieved.

[0057] The specific implementation process of the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the step processes shown in the accompanying drawings and embodiments can be executed in a computer system such as a set of computer executable instructions, and one or more of the exemplified method steps can be single or multiple or aggregated together and run in the computer system in the form of a software module or a virtual device. Moreover, although the exemplified step processes have time and sequential logic, the present invention is not limited to these steps in certain cases. They can be executed in an order different from that here. Under the guidance of the purpose of the present invention, simple changes or combinations of the execution order of these steps should also fall within the scope of the disclosure of the embodiments of the present invention.

[0058] The present invention is applied to storage systems using log-structured systems, such as key-value KV storage systems, file storage systems (such as F2FS), storage firmware (such as FTL) systems, etc. In an embodiment of the present invention, in order to obtain the input / output of a logic data block, that is, the IO update frequency (thereby predicting its hotness and coldness), the absolute time of the IO update is not recorded, but the IO update interval of the logic data block pointed to by the logic block address (LogicBlock Address, LBA) is calculated through a global self-incrementing IO counter, and the frequency and periodicity of accessing the data block are reflected according to the IO counting interval of the counter. This processing method can not only improve the system computing efficiency, but also obtain the adaptive effect to time.

[0059] Figure 1 A schematic flow chart of the steps of a method for dynamic grouping and garbage collection based on data update characteristics provided by an embodiment of the present invention. The method is applied to a storage system using a log write mode (i.e., recording modification / update operations in the form of appended logs). The method comprises:

[0060] Step 101. Update a data update information table reflecting the update frequency characteristics of the logical data block according to the address of the logical data block written by the IO write request; the data update information table records the latest n update interval values ​​of each logical data block, and the update interval is the difference between the global IO counter value between the current IO write request and the previous IO write request of the logical data block;

[0061] When receiving each IO write request, the storage node component that processes IO (such as the storage engine in distributed storage) will update the data update information table that reflects the update frequency and periodic characteristics of the logical data block according to the written logical data block address. For example, if an IO write request is to update the content of multiple logical data blocks, the logical data block address LBA corresponding to the written content can be obtained according to the write offset address and the amount of written data in the IO write request, and then the "data update information table" is updated according to the LBA address of the data block.

[0062] Table 1 below is a schematic diagram of the content and state of a data update information table at a certain moment in an embodiment of the present invention. To simplify the description, Table 1 only shows some logical data blocks, which are distinguished by the number of the LBA address field. For example, an LBA address field value of 1 represents the first logical data block in the storage system, 9 represents the ninth logical data block in the storage system, and so on.

[0063] Table 1

[0064]

[0065] The data update information table in the example of Table 1 includes 8 fields, namely:

[0066] The "LBA address" field is used to identify a logical data block in the storage system;

[0067] The "Latest Updated IO Counter Value" field is used to record the global IO counter value when the logical data block identified by the LBA address field was last written. The global IO counter is incremented by 1 each time an IO write request is received. For example, when the storage engine processes the 5568th IO write request, the current global IO counter value "5568" is recorded in the "Latest Updated IO Counter Value" field of the logical data block with LBA address 1.

[0068] The "Current Update Interval Recording Position" field is used to indicate in which update interval field from "Update Interval 1" to "Update Interval n" the "Update Interval Value" should be recorded. In the initialization state, the value of the "Current Update Interval Recording Position" field is initialized to 1, and is reset to 1 when the number of update interval groups n is reached.

[0069] The "Update Interval 1" to "Update Interval n" fields are used to record the global IO counter differences corresponding to the most recent n IO write requests of the logical data block. Each logical data block corresponds to n "Update Intervals", and the n update interval fields of the logical data block respectively record the difference in the global IO counter between the current IO write request and the previous IO write request for the logical data block. Among them, n is the number of update interval groups, which is configurable. In Table 1, n=6. The more update interval groups there are, the more accurate it is in measuring the access frequency and period of the logical data block. The update interval field is used in a cyclic manner, and the write position of the update interval value corresponding to the current IO write request is indicated by the "Record Position of This Update Interval" field.

[0070] As shown in Table 1, assuming that the current global IO counter value is 9999, after a new IO write request is received, the current global IO counter is incremented by 1 and becomes 10000. The logical data block written by the new IO write request includes the logical data block with LBA address 1, and the process of updating Table 1 is as follows:

[0071] First, read the "Last Updated IO Counter Value" field value 5568 and the "Current Update Interval Record Location" field value 3 of the logical data block with LBA address 1 in Table 1;

[0072] Then, the current global IO counter value 10000 is subtracted from the read "Last updated IO counter value" field value 5568 to get the update interval of 4432;

[0073] Then, according to the value 3 of the "Record Position of Current Update Interval" field, the update interval value 4432 is written into the "Update Interval 3" field, and then the value of the "Record Position of Current Update Interval" field is increased by 1 to 4;

[0074] Finally, the current global IO counter value 10000 is recorded in the "Latest Updated IO Counter Value" field of the logical data block with LBA address 1, overwriting the previous value 5568.

[0075] In the above manner, the storage engine can dynamically update the data update information table such as Table 1, thereby dynamically recording the update frequency of each logical data block and the periodic characteristics of the IO write request.

[0076] Step 102. Average the latest n update interval values ​​of each logical data block in the data update information table to obtain an update interval mean value, dynamically group each logical data block according to a preset update interval mean value threshold and the periodic characteristics of the n update interval values, and record the data grouping of each logical data block;

[0077] The present invention characterizes the degree of hotness or coldness of a data block by the mean update interval of each logical data block, that is, the smaller the mean update interval of the counter is, the higher the access frequency to the logical data block is, and the shorter the life cycle of the data stored therein is; on the contrary, the larger the mean is, the lower the access frequency to the logical data block is, and the longer the life cycle of the data stored therein is. However, the mean cannot reflect the periodic characteristics of the IO write request for the logical data block, so it is necessary to determine whether the IO write request for each logical data block has periodic characteristics based on the n update intervals recorded in the data update information table through a preset periodic feature recognition algorithm. There are many specific determination methods for determining whether an IO write request has periodic characteristics based on the update interval, such as the mean and variance method, the run method, etc. Since these determination methods are general methods, they are not described in detail.

[0078] The update interval mean threshold value and the correspondence between the update interval mean threshold value and the data block grouping can be configured through a system configuration file.

[0079] In the embodiment illustrated in Table 1, the logical data blocks are divided into three large groups based on the update interval mean and the periodic characteristics of IO write requests, namely, a frequent update group (C1 group), a periodic non-frequent update group (C2 group) and a random non-frequent update group (C3 group).

[0080] C1 grouping corresponds to the frequently updated hot data block grouping; since this grouping has a high update frequency, the periodicity of the update interval may not be considered;

[0081] C2 grouping corresponds to the infrequently updated (warm and cold) data block grouping with obvious periodic characteristics;

[0082] The C3 grouping corresponds to data block groupings that have no periodic characteristics (randomness) and are not frequently updated (the average update interval is greater than a preset threshold).

[0083] Figure 2 The coordinate diagram of the relationship between the data block grouping and the update interval mean and the update interval mean of the data block grouping in one embodiment of the present invention is shown in FIG.

[0084] like Figure 2 For example, the data blocks are grouped as follows:

[0085] C1 group: a frequently updated data block group whose average update interval is less than the first threshold T1 (1000).

[0086] C2 group: a periodic non-frequently updated data block group whose update interval mean is greater than or equal to the first threshold T1 and less than the sixth threshold T6 (6000) and whose update interval value presents periodic characteristics.

[0087] To further refine the grouping, Figure 2 In the example, the C2 group is further subdivided into multiple sub-data block groups. For example, the logical data blocks that are greater than or equal to the first threshold T1 (1000) and less than the threshold T2 (2000) and have periodic characteristics are divided into the C2-G1 group. The logical data blocks that are greater than or equal to the threshold T2 and less than the threshold T3 (3000) and have periodic characteristics are divided into the C2-G2 group, and so on, there are 5 groups in total, namely C2-G1, C2-G2, C2-G3, C2-G4, and C2-G5. The corresponding life cycle (represented by the mean value of the IO write request counter interval) of a group CX can be expressed as CX-T, for example, C1-T=1000, C2-G1-T=2000, C2-G5-T=6000.

[0088] C3 group: It is a group composed of logical data blocks that cannot be classified into C1 group or C2 group according to the characteristics of the average update interval. For example, if the update interval values ​​of logical data blocks vary greatly and have no periodic characteristics, such logical data blocks can be classified into C3 group.

[0089] In one embodiment of the present invention, in order to reflect the recency of the update interval mean and the time when the IO write request occurs, when calculating the update interval mean of each logical data block, the update interval mean is calculated by weighted average. The calculation method is to assign a higher weight to the update interval value corresponding to the IO write request with a closer occurrence time, and the sum of the weights of n update intervals is 1. Each update interval is multiplied by its corresponding weight and then the average value is calculated to obtain the weighted update interval mean of the logical data block. The weighted update interval mean can better reflect the frequency and periodicity characteristics of the most recent IO write request, so as to more effectively and dynamically adjust the grouping of each logical data block.

[0090] Step 103: Dynamically adjust the number of segments allocated to each data group according to the percentage of the sum of the update interval averages of all logical data blocks in the data group to the sum of the update interval averages of all logical data blocks.

[0091] For example, assuming that the storage system has a total of S data segments, the sum of the update interval means of all logical data blocks in the storage system is K, and the sum of the update interval means of all logical data blocks in the data group CX is CX-u, then the number of segments obtained by the data group CX is recorded as Conf-CX:

[0092]

[0093] The calculation result can be rounded off. The value range of Conf-CX is greater than or equal to 1 and less than S.

[0094] Step 104: Configure different garbage collection strategies for segments in different data groups.

[0095] For example, for the aforementioned hot data group C1 and the random infrequently updated data group C3, a greedy algorithm may be used to perform garbage collection on the data blocks in the segments contained in the data groups.

[0096] For the sub-data groups in data group C2, since the logical data they contain has similar periodic characteristics, the benefit-cost algorithm can be used for all data groups in the entire C2 data group to perform garbage collection on the data blocks in the segment. Because the data blocks contained in each Segment have a similar life cycle, its value is assumed to be T (the T value can be recorded as the metadata of the Segment when the data is written); at the same time, it is assumed that the effective data occupancy rate of the Segment is U (0≤U≤1, a dynamically changing value). The benefit cost number of each Segment is defined as (1-U)*T. At this point, the benefit cost values ​​of all Segments in the group can be calculated, and the P Segments with the largest values ​​are selected for garbage collection.

[0097] Based on the above disclosed technical solution, when the storage engine receives an IO write request, it needs to first update the "data update information table", and then write the data into the data segment of the corresponding data group according to the data group to which the logical data block belongs. In order to avoid too frequent adjustment of the data grouping of the logical data block, the storage system can set a grouping adjustment period TC (such as 5 minutes, the value of which is configurable), and periodically adjust the grouping configuration of the data block according to TC, so as to dynamically adapt to different IO load models.

[0098] Based on the above-mentioned solution provided by the present invention for dynamically grouping and garbage collecting logical data blocks according to data update characteristics, the present invention can allocate segment numbers and garbage collection strategies that match the grouping characteristics to different types of data groups, thereby optimizing the overall write amplification factor WAF of the storage system. By optimizing WAF, technical effects such as increasing the service life of the hard disk, improving write performance, and improving system stability can be achieved.

[0099] Figure 3 A schematic diagram of a device for dynamic grouping and garbage collection based on data update characteristics provided by an embodiment of the present invention. The device 300 is applied to an electronic device of a storage node in a storage system using a log structure system, and the device 300 includes:

[0100] The information table updating module 310 is used to update the data update information table reflecting the update frequency characteristics of the logical data block according to the address of the logical data block written by the IO write request; the data update information table records the latest n update interval values ​​of each logical data block, and the update interval value is the difference between the global IO counter value between the current IO write request and the previous IO write request of the logical data block;

[0101] The dynamic grouping module 320 is used to average the latest n update interval values ​​of each logical data block in the data update information table to obtain an update interval mean value, dynamically group each logical data block according to a preset update interval mean value threshold and the periodic characteristics of the n update interval values, and record the data grouping of each logical data block. The dynamic grouping module 320 can adjust the grouping of logical data blocks in the storage system based on the dynamically updated data update information table according to a preset period.

[0102] A dynamic segment allocation module 330, for dynamically adjusting the number of segments allocated to each data group according to the percentage of the sum of the update interval means of all logical data blocks in the data group to the sum of the update interval means of all logical data blocks;

[0103] The recycling strategy setting module 340 is used to configure different garbage collection strategies for segments in different data groups. The module sets a garbage collection strategy matching the group to which the segment belongs based on the adjusted or allocated segment.

[0104] Preferably, the information table updating module 310 records the latest n update interval values ​​in the data update information table in a circular manner through n fields; and also records the position of the update interval value that the current IO write request should write through a position field;

[0105] When calculating the update interval mean value of each logical data block, the information table update module 310 calculates the update interval mean value in a weighted average manner, and the update interval value corresponding to the IO write request with a closer occurrence time has a higher weight.

[0106] Preferably, the dynamic grouping module 320 further includes:

[0107] A periodic feature judgment submodule, used to judge whether the IO write request for each logical data block has a periodic feature based on the n update interval value recorded in the data update information table by using a preset periodic feature recognition algorithm;

[0108] The data grouping submodule is used to divide the frequently updated logical data blocks into the frequently updated group (C1 group) based on the update interval mean threshold; divide the logical data blocks with the update interval mean within the same threshold range into the periodic non-frequent update group (C2 group) based on the update interval mean threshold; and divide all logical data blocks that do not belong to the frequently updated group (C1 group) or the periodic non-frequent update group (C2 group) into the random non-frequent update group (C3 group).

[0109] Preferably, the recycling strategy setting module 340 configures a greedy algorithm to perform garbage collection on data blocks in segments contained in the frequently updated group (C1 group) and the randomly infrequently updated group (C3 group); and configures a benefit-cost algorithm to perform garbage collection on data blocks in segments contained in the periodically infrequently updated group (C2 group).

[0110] Figure 4A schematic diagram of the structure of an electronic device for implementing the method of dynamic grouping and garbage collection based on data update features provided by one embodiment of the present invention, the device 400 includes: a processor 410 such as a central processing unit (CPU), a communication bus 420, a communication interface 440 and a memory 430. The processor 410 and the memory 430 can communicate with each other through the communication bus 420. The memory 430 stores a computer program, and when the computer program is executed by the processor 410, the functions of one or more steps in the method of dynamic grouping and garbage collection based on data update features provided by the present invention can be implemented in the form of a software module. When the program for implementing the method provided by the present invention is loaded into the memory 430 in the device 400, it is equivalent to running the program in the device 400 as follows. Figure 3 An exemplary apparatus 300 is shown.

[0111] Memory refers to a device based on a certain storage medium for storing computer programs and / or data. It can be a volatile memory (VM, often called memory) or a non-volatile memory (NVM). Memory refers to an internal memory that directly exchanges data with the processor. It can read and write data at any time and at a very fast speed. It is used as a storage medium for temporary data of the operating system and other running programs. Memory can be synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), etc. Non-volatile memory refers to a memory that uses a persistent storage medium. It has the characteristics of large capacity and persistent data storage. It can be a storage class memory (SCM), a solid state disk (SSD), a NAND flash memory, a disk, etc. SCM is the industry's general term for new storage media between memory and flash memory. It is a composite storage technology that combines persistent storage characteristics and memory characteristics. Its access speed is slower than DRAM and faster than SSD hard disks.

[0112] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0113] It should be appreciated that embodiments of the present invention may be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory (or non-persistent) memory. The method may be implemented in a computer program using standard programming techniques, including a non-transitory storage medium configured with a computer program, wherein the storage medium configured in this way causes the computer to operate in a specific and predefined manner. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if necessary, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose. In addition, the operation of the process described in the present invention may be performed in any suitable order, unless the present invention otherwise indicates or otherwise clearly contradicts the context. The process described in the present invention (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0114] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described in the present invention includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0115] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for dynamic grouping and garbage collection based on data update characteristics, characterized in that: The method is applied to a storage system using a log structure system, and the method includes: A data update information table reflecting the update frequency characteristics of the logical data block is updated according to the logical data block address written by the input / output IO write request; the data update information table records the latest n update interval values ​​of each logical data block, and the update interval value is the difference between the global IO counter value of the current IO write request and the previous IO write request of the logical data block; The update interval values ​​of the latest n times of each logical data block in the data update information table are averaged to obtain the update interval mean value, each logical data block is dynamically grouped according to the preset update interval mean value threshold and the periodic characteristics of the n update interval values, and the data grouping of each logical data block is recorded; Dynamically adjust the number of segments allocated to each data group according to the percentage of the sum of the update interval averages of all logical data blocks in the data group to the sum of the update interval averages of all logical data blocks; Configure different garbage collection policies for segments within different data groups.

2. The method according to claim 1, characterized in that In the data update information table, the update interval values ​​of the latest n times are recorded in a circular manner through n fields; and the position of the write update interval value corresponding to the current IO write request is recorded through a position field; When calculating the update interval mean value of each logical data block, a weighted average method is used to calculate the update interval mean value, and the update interval value corresponding to the IO write request with a closer occurrence time has a higher weight.

3. The method according to claim 1, characterized in that By using a preset periodic feature recognition algorithm and based on the n update interval values ​​recorded in the data update information table, it is determined whether the IO write request for each logical data block has a periodic feature; Based on the update interval mean threshold, the frequently updated logical data blocks are divided into the frequently updated group; Based on the update interval mean value threshold, logical data blocks whose update interval mean values ​​are within the same threshold range are divided into periodic infrequent update groups; All logical data blocks that do not belong to the frequent update group or the periodic infrequent update group are divided into the random infrequent update group.

4. The method according to claim 3, characterized in that Based on multiple update interval mean thresholds, the logical data blocks in the periodic infrequent update group are further divided into multiple sub-data groups.

5. The method according to claim 3, characterized in that: For the frequently updated group and the random infrequently updated group, a greedy algorithm is configured to perform garbage collection on the data blocks in the segments contained in the data group; For data blocks in segments contained in periodic infrequent update groups, a benefit-cost algorithm is configured for garbage collection.

6. A device for dynamic grouping and garbage collection based on data update characteristics, characterized in that: The device is applied to a storage system adopting a log structure system, and the device includes: An information table updating module is used to update a data update information table reflecting the update frequency characteristics of the logical data block according to the address of the logical data block written by the IO write request; the data update information table records the latest n update interval values ​​of each logical data block, and the update interval value is the difference between the global IO counter value between the current IO write request and the previous IO write request of the logical data block; A dynamic grouping module is used to average the latest n update interval values ​​of each logical data block in the data update information table to obtain an update interval mean value, dynamically group each logical data block according to a preset update interval mean value threshold and the periodic characteristics of the n update interval values, and record the data grouping of each logical data block; A dynamic segment allocation module, used to dynamically adjust the number of segments allocated to each data group according to the percentage of the sum of the update interval averages of all logical data blocks in the data group to the sum of the update interval averages of all logical data blocks; The recycling strategy setting module is used to configure different garbage collection strategies for segments in different data groups.

7. The device according to claim 6, characterized in that The information table update module records the latest n update interval values ​​in the data update information table in a circular manner through n fields; and also records the position of the write update interval value corresponding to the current IO write request through a position field; When calculating the update interval mean value of each logical data block, the information table update module calculates the update interval mean value in a weighted average manner, and the update interval value corresponding to the IO write request with a closer occurrence time has a higher weight.

8. The device according to claim 6, characterized in that The dynamic grouping module comprises: A periodic feature judgment submodule, used to judge whether the IO write request for each logical data block has a periodic feature based on the n update interval value recorded in the data update information table by using a preset periodic feature recognition algorithm; The data grouping submodule is used to divide the frequently updated logical data blocks into the frequently updated group (C1 group) based on the update interval mean threshold; divide the logical data blocks with the update interval mean within the same threshold range into the periodic non-frequent update group (C2 group) based on the update interval mean threshold; and divide all logical data blocks that do not belong to the frequently updated group (C1 group) or the periodic non-frequent update group (C2 group) into the random non-frequent update group (C3 group).

9. The device according to claim 8, characterized in that The recycling strategy setting module configures a greedy algorithm to perform garbage collection on the data blocks in the segments contained in the frequent update group (C1 group) and the random infrequent update group (C3 group); and configures a benefit-cost algorithm to perform garbage collection on the data blocks in the segments contained in the periodic infrequent update group (C2 group).

10. An electronic device, characterized in that: It includes a processor, a communication interface, a storage medium and a communication bus, wherein the processor, the communication interface and the storage medium communicate with each other via the communication bus; Storage medium for storing computer programs; A processor, for implementing the method according to any one of claims 1 to 5 when executing a computer program stored on a storage medium.

Images