Data storage method and device and medium

Abstract

The invention discloses a data storage method, a data storage device and a medium, relates to the technical field of data storage, and provides a data storage method aiming at the write-in performance bottleneck caused by dirty data brushing under a cache pool when an existing distributed storage system stores data in a layered manner based on an erasure correction form. After to-be-written data is received, a data part with aligned stripes is directly written into a data pool. On one hand, data reading processing in the writing process is avoided, and the data writing speed of the data pool is increased. And on the other hand, the data pool is written by taking a group of stripes aligned with the stripes as a unit, which is equivalent to the realization of aggregation of the small block random IO, and the situation that the pressure of the small block random IO writing on the data pool becomes the bottleneck of the writing performance of the data pool is avoided. In conclusion, the data writing performance of the data pool when the data pool stores the data in the EC form is greatly improved, and the bottleneck problem of the writing performance of the data pool can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a data storage method, device and medium. Background Art

[0002] In a distributed storage system, tiered storage is achieved by dividing different storage media into different storage pools. Since the space of the cache pool is generally much smaller than that of the data pool, when data is written to the cache pool, if there is insufficient remaining space in the cache pool, the data in the cache pool needs to be flushed to the data pool. The data in the cache pool that needs to be flushed to the data pool is dirty data.

[0003] If dirty data is all random small-block input and output data (IO), then when flushing data, the pressure on the storage medium of the data pool is also the pressure of small-block random write IO. Especially when the cache pool stores data in the form of copies, and the data pool stores data in the form of erasure code (EC), the EC algorithm needs to recalculate the check block every time data is written. Therefore, for modification writes, if the data written this time is not full stripe data, it is necessary to first read the other stripe data of the corresponding stripe from the data pool, merge them into a complete stripe, and then calculate the check block to complete the writing of data to the data pool. At this time, the small-block IO random write performance of the data pool determines the flushing speed of dirty data, and then determines the front-end business write performance.

[0004] Therefore, technicians in this field are in urgent need of a data storage method to solve the write performance bottleneck of flushing dirty data from a cache pool when the current distributed storage system stores data in a layered manner based on erasure correction. Summary of the invention

[0005] The purpose of the present invention is to provide a data storage method, device and medium to solve the write performance bottleneck of flushing dirty data in a cache pool when the current distributed storage system stores data in a layered manner based on erasure correction.

[0006] In order to solve the above technical problems, the present invention provides a data storage method, which is applied to a distributed storage system, wherein the distributed storage system includes a cache pool and a data pool, and the method includes:

[0007] When receiving the data to be written, performing a stripe alignment check on the data to be written and the dirty data currently stored in the buffer pool as a whole to determine the stripe-aligned portion of the data to be written and the dirty data, and the non-stripe-aligned portion of the data to be written;

[0008] Writing the data to be written and a portion of the dirty data whose stripes are aligned into the data pool for storage;

[0009] Part of the data to be written that is not stripe-aligned is written into the buffer pool as new dirty data for storage.

[0010] In a possible embodiment, it further includes:

[0011] Divide the dirty data into stripes, and use stripe data belonging to the same stripe as a group of data objects to be flushed;

[0012] According to the different data volumes, each of the data objects to be flushed is added to the corresponding least recently used management queue; wherein different least recently used management queues are used to manage the data objects to be flushed with different data volumes, and each least recently used management queue manages the data objects to be flushed in the queue with the least recently used strategy;

[0013] When the dirty data needs to be flushed to the data pool, the data objects to be flushed are preferentially selected from the least recently used management queue with a large management data volume for flushing.

[0014] In a possible embodiment, each time after the dirty data is flushed, the method further includes:

[0015] Get the current water level of the cache pool;

[0016] Determine the corresponding refresh cycle according to the water level interval to which the current water level belongs; wherein the water level interval is a plurality of intervals pre-divided for the water level of the buffer pool; and the higher the water level corresponding to the water level interval, the shorter the corresponding refresh cycle;

[0017] After waiting for the time corresponding to the refresh cycle, the next dirty data refresh action is triggered.

[0018] In a possible embodiment, the water level of the cache pool is divided into a plurality of water level intervals according to a plurality of pre-set water level nodes, and each of the water level nodes uniquely corresponds to a level down refresh cycle;

[0019] Then, determining the corresponding next refresh cycle according to the water level interval to which the current water level belongs includes:

[0020] Determine an intermediate coefficient according to two groups of water level nodes corresponding to the upper limit water level value and the lower limit water level value of the water level interval to which the current water level belongs and the refresh delay;

[0021] Determine an intermediate result according to the current water level and the first coefficient;

[0022] The refresh cycle corresponding to the current water level is determined according to the intermediate result and the refresh delay corresponding to the lower limit water level value of the water level interval to which the current water level belongs.

[0023] In a possible embodiment, whenever a data operation request is received, the method further includes:

[0024] Updating the popularity information of the data object targeted by the data operation request;

[0025] The heat information is a type of data object metadata and is stored in the cache pool.

[0026] In a possible embodiment, after updating the popularity information of the data object targeted by the data operation request, the method further includes:

[0027] If the data object is determined to be hot data, a portion of the data in the data object corresponding to the portion stored in the data pool is determined as target data;

[0028] Migrating the target data from the data pool back to the cache pool for storage;

[0029] If the heat information of the data object is not updated within a preset period of time, the data object is regarded as the dirty data.

[0030] In a possible embodiment, each time after the dirty data is flushed, the method further includes:

[0031] Get the total time required to complete the flushing of the dirty data.

[0032] If the total time consumption exceeds the preset time consumption threshold, the position of the down-sweep permission mark is set to a pause state;

[0033] Among them, the down-brush enable flag includes: an enable state and a pause state, and the down-brush enable flag defaults to the enable state; when the down-brush enable flag is in the enable state, the down-brush of the dirty data is allowed, and when the down-brush enable flag is in the pause state, the down-brush of the dirty data is prohibited.

[0034] In order to solve the above technical problems, the present invention further provides a data storage device, which is applied to a distributed storage system. The distributed storage system includes a cache pool and a data pool. The device includes:

[0035] A checking module, for, when receiving data to be written, performing a stripe alignment check on the data to be written and the dirty data currently stored in the cache pool as a whole, so as to determine the stripe-aligned portion of the data to be written and the dirty data, and the non-stripe-aligned portion of the data to be written;

[0036] A flush module, used for writing the data to be written and the stripe-aligned part of the dirty data into the data pool for storage;

[0037] The cache module is used to write the non-stripe-aligned part of the data to be written as new dirty data into the cache pool for storage.

[0038] In order to solve the above technical problems, the present invention further provides a data storage device, comprising:

[0039] Memory for storing computer programs;

[0040] A processor is used to implement the steps of the data storage method described above when executing the computer program.

[0041] In order to solve the above technical problem, the present invention also provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data storage method described above are implemented.

[0042] The present invention provides a data storage method, which performs a stripe alignment check on the data to be written when data needs to be written; if there is a stripe-aligned data part in the data to be written (which can be full-score stripe data composed entirely of stripe data in the data to be written, or full-score stripe data pieced together from stripe data of the data to be written and dirty data in a cache pool), this part of the stripe-aligned data is directly written to the data pool; and the non-stripe-aligned part of the data to be written is written to the cache pool as a dirty data cache; when new data is subsequently written, it can participate in the subsequent stripe alignment check until the stripe-aligned full-score stripe data is gathered and then written to the data pool as a whole in the form of the full-score stripe data.

[0043] Based on the data writing method of this method, every time data is written to the data pool, data is written as stripe-aligned full-strip data. In this way, on the one hand, there is no need to read the missing part of the non-stripe-aligned data from the data pool to calculate the check block, which avoids the data processing in the writing process and improves the writing speed of data into the data pool. On the other hand, it also guarantees the amount of data written to the data pool at one time. When small IO data is received (that is, the data to be written cannot make up a full stripe-aligned full-strip data), it will be cached in the cache pool until a certain amount of data is accumulated (that is, a group of full stripe-aligned full-strip data) and then written to the data pool. This process realizes the write aggregation of small IO data, which can effectively alleviate the pressure of small IO writes on the data pool and improve the small IO write performance of the data pool. The performance optimization of writing data to the data pool from the above two aspects can effectively solve the performance bottleneck problem that exists when writing data to the current data pool.

[0044] The data storage device and non-volatile storage medium provided by the present invention correspond to the above method and have the same effects as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 An application architecture diagram of a distributed storage system provided by an embodiment of the present invention;

[0047] Figure 2 A flowchart of a data storage method provided by an embodiment of the present invention;

[0048] Figure 3 A flowchart of a write request processing provided by an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of a dirty data flushing cycle provided by an embodiment of the present invention;

[0050] Figure 5 A flowchart of a read request processing provided by an embodiment of the present invention;

[0051] Figure 6 A structural diagram of a data storage device provided by an embodiment of the present invention;

[0052] Figure 7 A structural diagram of another data storage device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The core of the present invention is to provide a data storage method, device and medium.

[0055] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0056] Currently, tiered storage has become a common storage method. For example, in a distributed storage system, a storage pool (cache pool) composed of high-speed but expensive storage devices such as solid state disks (SSDs) is used as a cache layer, and a storage pool (data pool) composed of relatively low-speed but cheap storage devices such as hard disk drives (HDDs) is used as a data layer.

[0057] In addition, for data storage systems such as distributed storage systems, data redundancy strategies are also adopted to ensure storage reliability and data availability. Common data redundancy strategies include replication and erasure coding (EC). Since the performance and cost of the cache pool and the data pool are different, the data redundancy methods used by the cache pool and the data pool are generally different. The cache pool generally stores data in the form of replicas, and the data pool generally stores data in the form of EC.

[0058] Based on the above background, when an external device (such as a client) writes data to a distributed storage system, it generally chooses whether to store the data in a cache pool or a data pool based on the access frequency of the data. If the access frequency of the data is high, the data is stored in a cache pool with high performance but small capacity. If the access frequency of the data is low, the data is stored in a data pool with low performance but large capacity (high, low, and large are relative concepts between cache pools and data pools).

[0059] In this way, the following problems will arise:

[0060] 1. The physical storage medium on which the data pool is based has limited performance, so it is more likely to hit a performance bottleneck when writing random small-block input and output (IO). At this time, the performance of the data pool writing random small-block IO restricts the data pool's data writing performance, and the data pool's data writing performance in turn restricts the front-end business writing performance of the distributed storage system.

[0061] 2. The data in the data pool is stored in EC form. The principle of EC is to divide the original data into k parts, and then calculate m check blocks based on these k data blocks based on the EC algorithm. After that, based on any k of these k+m EC data, the complete original data can be restored to achieve data redundancy. Therefore, when writing EC data to the data pool, it is necessary to write these k+m EC data.

[0062] However, the data written by the client is not in EC format, and may even contain incomplete original data (for example, random small blocks of IO are written). In this case, it is necessary to read the missing original data from the data pool before writing EC data to the data pool. After piecing together the complete original data, EC calculation is performed, and finally the calculated EC data is written to the data pool. It can be seen that the data pool may need to perform a data read step before writing data, which greatly affects the EC write speed of the data pool.

[0063] To solve the above problems, the present invention provides a data storage method, which is applied to a distributed storage system. Figure 1 As shown, it includes a cache pool and a data pool. Figure 2 As shown, including:

[0064] S11: When receiving the data to be written, the data to be written and the dirty data stored in the current buffer pool are subjected to a stripe alignment check as a whole to determine the stripe-aligned portion of the data to be written and the dirty data, and the non-stripe-aligned portion of the data to be written.

[0065] S12: writing the stripe-aligned part of the data to be written and the dirty data into the data pool for storage.

[0066] S13: writing part of the data to be written that is not stripe-aligned into the buffer pool as new dirty data for storage.

[0067] The above-mentioned data to be written refers to the data that needs to be written to the data pool in the end. For data that only needs to be written to the cache pool but not the data pool, it does not involve the problem of improving the data pool writing performance bottleneck targeted by this method, so it is not within the scope of discussion of this method. In addition, there is no order relationship between step S12 and step S13, and they can be implemented in parallel or in any order.

[0068] In addition, the above-mentioned dirty data refers to the data in the cache pool that needs to be flushed to the data pool. That is, this part of the data is first cached in the cache pool for some reason, but the ultimate goal is to be written to the data pool and stored in the form of EC. It should be noted that this embodiment does not care when the dirty data is flushed to the data pool. As long as it is determined that a certain part of the data in the cache pool needs to be flushed to the data pool, this part of the data is the dirty data in the above steps. Dirty data is determined by a certain part of the data in the pre-specified cache pool that needs to be flushed to the data pool. Based on the above steps S11~S13, each time the data to be written is received, part of the data to be written that is not directly written to the data pool due to non-strip alignment will also be written to the data pool as new dirty data. It is easy to understand that the new dirty data does not affect the old dirty data, and the newly written dirty data and the original old dirty data together constitute the current dirty data in the cache pool.

[0069] Further, for step S11. Based on the principle of the above EC technology, it can be known that EC will split the original data block into k data blocks, and calculate m check blocks based on the k data blocks EC, and the k data blocks and the m data blocks are collectively referred to as EC data. And each data block or check block is called a stripe, and a group of stripes (k+m) constitutes a stripe. Therefore, the stripe alignment check in step S11 is to check whether there is stripe-aligned EC data in the data to be written. More specifically, the above check is to check whether there are any k EC data (more than k) in a stripe with a total of k+m EC data. If it is available, it means that the complete original data can be restored based on these k EC data, and the complete k+m EC data can be calculated based on EC (that is, it is regarded as a full stripe). This part of the data (that is, the stripe-aligned part of the data to be written and the dirty data) can be directly written to the data pool as in step S12, without the need to perform the data reading process mentioned in the above question 2. As for other data (ie, the non-stripe-aligned part of the to-be-written data), as in step S13, it is written into the buffer pool as new dirty data and waits for the next stripe alignment check.

[0070] In addition, it should be noted that the stripe alignment check performed in step S11 is performed on the data to be written and the dirty data stored in the current buffer pool as a whole, rather than performing stripe alignment checks on the data to be written and the dirty data stored in the current buffer pool separately. That is, if the data to be written is regarded as data set A, and the dirty data stored in the current buffer pool is regarded as data set B, then step S11 is to perform a stripe alignment check on the set C of data set A + data set B (A∪B=C). The check results may be as follows:

[0071] A. There is stripe-aligned data in the data to be written (excluding dirty data);

[0072] In this case, the data to be written itself contains any k (or more than k) stripes that can be put together in a stripe. Based on these k stripes, a full stripe can be calculated, which is regarded as full stripe data. As shown in step S12, this part of the stripe alignment data in the data to be written is directly written to the data pool.

[0073] B. The data to be written is not stripe-aligned, but the data required to satisfy stripe alignment can be gathered together with dirty data;

[0074] This situation is that the data to be written itself cannot make up any k stripes in a stripe. However, the dirty data in the cache pool may include other stripes of the stripe. After combining with the dirty data, any k (or more than k) stripes under this stripe can be made up. At this time, as in step S12, the stripe corresponding to the data to be written and the stripe corresponding to the dirty data of this stripe need to be directly written to the data pool.

[0075] C. The data to be written is not stripe-aligned, and after combining with dirty data, the data required to satisfy stripe alignment cannot be gathered;

[0076] This situation is that after the data to be written and the dirty data are considered as a whole, some data still cannot make up the k stripes required to calculate the full stripe. At this time, as in step S13, this part of the data needs to be stored in the cache pool as dirty data and wait for the next stripe alignment check. However, since the non-stripe-aligned data in the dirty data has already been written into the cache pool, and it is dirty data itself and does not need to be rewritten. Therefore, in step S13, only the non-stripe-aligned part of the data to be written needs to be written into the cache pool and used as new dirty data.

[0077] It should also be noted that based on the above dirty data update mechanism, that is, except for the custom specified dirty data, it is impossible to have only dirty data to complete the k stripes that meet the stripe calculation. In other words, it is impossible for the dirty data to have complete stripe alignment data. Only by relying on the data to be written received later, it is possible to make part of the dirty data complete the k stripes of stripe alignment and flush it.

[0078] In addition, the implementation of this method needs to rely on the cache pool and data pool of the distributed storage system. In a possible application scenario, the hardware architecture and principle diagram for completing a data write request are as follows: Figure 3As shown: the client's write request is first sent to the primary object storage device (OSD) in the cache pool; the primary OSD performs a stripe alignment check on the data to be written, and sends the non-stripe aligned part to other OSDs in the cache pool to complete the writing; the primary OSD also forwards the stripe aligned part of the data to be written and the dirty data to the primary OSD in the data pool, which completes the EC data distribution to realize the data writing of the data pool.

[0079] It can be seen that the device that plays a major role in the above write processing flow is the main OSD of the cache pool. Therefore, this method can specifically use the main OSD in the cache pool as the execution subject. Among them, OSD is the physical device that constitutes the storage pool (cache pool / data pool) in the distributed storage system. For example, devices such as hosts that deploy OSD services and storage media such as hard disks. The main OSD is an OSD device selected from the many OSDs that constitute the storage pool, which implements data distribution to ensure data consistency.

[0080] In summary, the data storage method provided by the present invention performs a stripe alignment check on the data to be written and the dirty data in the cache pool as a whole after receiving the data to be written, and directly writes the stripe-aligned data portion into the data pool. In this way, on the one hand, it can be avoided that when the data pool writes data in the form of EC, before writing the non-stripe-aligned data, it is necessary to read other data from the data pool to make up a striped k stripes to meet the writing premise, thereby avoiding the data reading process in the writing process and improving the speed of writing data into the data pool. On the other hand, in this method, writing to the data pool is performed in units of a group of stripe-aligned stripes, which is equivalent to realizing the aggregation of small-block random IO, avoiding the pressure of small-block random IO writing on the data pool becoming a bottleneck for the writing performance of the data pool. In summary, the above two aspects greatly improve the data writing performance of the data pool when storing data in the form of EC, and can effectively solve the writing performance bottleneck problem of the data pool.

[0081] In addition, in this method, after the non-stripe-aligned part of the data to be written is written to the cache pool as dirty data, the dirty data can also participate in the stripe alignment check of the data to be written later. If it can achieve stripe alignment with the data in the data to be written, it will be written to the data pool together. In this way, a dirty data flushing mechanism is implemented, which avoids the continuous accumulation of non-stripe-aligned part of the data to be written in the cache pool, and then the dirty data fills up the cache pool space.

[0082] On the other hand, in the above embodiment, the non-stripe-aligned part of the data to be written is written into the buffer pool as dirty data, and after receiving new data to be written, if the stripe is aligned, it is flushed to the data pool to complete the EC data placement. However, in actual applications, some dirty data may not be able to be aligned into stripes for a long time, which leads to the problem of occupying the buffer pool storage space for a long time.

[0083] To solve this problem, the dirty data in the cache pool can be managed by the least recently used (Least Recently Used, LRU) strategy. This embodiment provides a further dirty data flushing management solution based on the LRU strategy, and the method also includes:

[0084] S21: Divide the dirty data into stripes, and treat the stripe data belonging to the same stripe as a group of data objects to be flushed.

[0085] S22: adding each data object to be flushed to a corresponding least recently used management queue according to different data volumes.

[0086] Different least recently used management queues are used to manage data objects to be flushed with different data volumes, and each least recently used management queue manages the data objects to be flushed in the queue using a least recently used strategy.

[0087] S23: When dirty data needs to be flushed to the data pool, data objects to be flushed are preferentially selected from the least recently used management queue with a large management data volume for flushing.

[0088] For example, assume that there are 4 different LRU queues a~d, where queue a manages data objects to be flushed with a data size of 1M~2M, queue b manages data objects to be flushed with a data size of 2M~4M, queue c manages data objects to be flushed with a data size of 4M~8M, and queue d manages data objects to be flushed with a data size of 8M~16M. When dirty data needs to be flushed, the data objects to be flushed are preferentially selected from queue d based on the LRU strategy for flushing. If there are no data objects to be flushed in queue d, the data objects to be flushed are preferentially selected from queue c, and so on.

[0089] As can be seen from the above, this embodiment manages data objects (dirty data) with different data volumes to be flushed through multiple different LRU queues. When dirty data needs to be flushed from the cache pool to the data pool, the data objects to be flushed are preferentially selected from the LRU queue with a large data volume to be flushed, so that the amount of dirty data flushed is as large as possible, thereby freeing up more cache pool space.

[0090] Furthermore, this embodiment does not limit how many data objects to be flushed at one time. It can be flushed one at a time or multiple at a time. After determining an LRU queue, how the LRU queue selects the data objects to be flushed is the content of the LRU strategy, and this embodiment does not make special requirements for this. However, for the time interval between two dirty data flushes, this embodiment provides a possible implementation plan. After each flush of the dirty data is completed, the method also includes:

[0091] S31: Get the current water level of the cache pool.

[0092] The water level of the cache pool (Cache) is the ratio of the currently used space in the cache pool to the total space.

[0093] S32: Determine the corresponding next refresh cycle according to the water level interval to which the current water level belongs.

[0094] The water level interval is a plurality of intervals pre-divided for the water level of the buffer pool, and the higher the water level corresponding to the water level interval, the smaller the corresponding flushing cycle.

[0095] S33: After waiting for a time corresponding to the refresh cycle, the next dirty data refresh action is triggered.

[0096] It should be noted that this embodiment does not limit the number of water level intervals, and the interval sizes of each water level interval can be equal or unequal. However, it is easy to know that in order to ensure that a refresh cycle can be uniquely determined at any water level in the cache pool, there should be no overlapping range between the water level intervals, and all restricted water level intervals should completely cover the entire range of the water level.

[0097] In one possible example, Figure 4 As shown, Figure 4 The entire range of the buffer pool water level is divided into 6 water level intervals (0~low, low~medium, medium~high, high~nearly full, nearly full~full, full~100) based on the 5 pre-set water level points (from low to high, they are low, medium, high, nearly full, and full). These 6 water level intervals use different delay durations as the flushing cycle, and the corresponding flushing cycle becomes shorter and shorter from low to high as the water level corresponding to the water level interval. That is, the less remaining space in the buffer pool, the higher the water level of the buffer pool, the shorter the corresponding flushing cycle, the higher the frequency of triggering the dirty data flushing action, and the faster the storage space of the buffer pool is released.

[0098] In addition, for different water levels in the same water level range, the same or different flushing cycles can be used. This embodiment provides a solution for determining the flushing cycle for the latter method:

[0099] like Figure 4 As shown, the water level of the cache pool is divided into multiple water level intervals according to multiple pre-set water level nodes, and each water level node uniquely corresponds to a level down refresh cycle. Then the above step S32 specifically includes:

[0100] S321: Determine the intermediate coefficient according to two groups of water level nodes and the down refresh delay corresponding to the upper limit water level value and the lower limit water level value of the water level interval to which the current water level belongs.

[0101] S322: Determine an intermediate result according to the current water level and the first coefficient.

[0102] S323: Determine the refresh cycle corresponding to the current water level according to the intermediate result and the refresh delay corresponding to the lower limit water level value of the water level interval to which the current water level belongs.

[0103] Also Figure 4 The above steps of this embodiment are described by way of example:

[0104] Assuming that the current water level ratio belongs to the water level interval (medium, high), the two water level nodes determined in step S321 are the lower limit water level value "medium" and the upper limit water level value "high". The level down refresh cycle corresponding to "medium" is "medium water level delay", and the level down refresh delay corresponding to "high" is "high water level delay". Based on the two coordinate points (medium, medium water level delay) and (high, high water level delay), the slope between the two coordinate points can be determined, that is, the first coefficient in the above step S11.

[0105] Afterwards, based on the first coefficient determined above and the horizontal intercept of the current water level in this water level interval, the vertical intercept of the current water level in this water level interval can be determined, that is, the intermediate result in the above step S12.

[0106] Finally, by adding the vertical intercept of the water level node in the corresponding direction to the intermediate result, we can get the down-brush cycle corresponding to any water point in the water level interval (medium, high).

[0107] In short, the refresh cycle method provided in this embodiment can be expressed by the following formula:

[0108] T=(min _delay – max _delay) / (max _ ratio – min _ ratio)*(max _ratio-ratio) + max _delay;

[0109] Among them, T is the refresh cycle; ratio is the current water level; max_ratio and min_ratio are the upper and lower water level values ​​of the water level range to which the current water level ratio belongs; max_delay and min_delay are the refresh delays corresponding to max_ratio and min_ratio respectively.

[0110] As can be seen from the above, this embodiment can uniquely determine a corresponding flush cycle at any water level of the buffer pool, and the size of the flush cycle is negatively correlated with the water level of the buffer pool. That is, the higher the water level of the buffer pool, the shorter the flush cycle, the faster the frequency of flushing dirty data from the buffer pool to the data pool, and the faster the storage space of the buffer pool can be released.

[0111] On the other hand, based on the mechanism for managing dirty data flushing based on the LRU strategy provided in the above embodiment, this embodiment also provides a corresponding implementation scheme:

[0112] Whenever a data operation request is received, the method further comprises:

[0113] S24: Update the popularity information of the data object targeted by the data operation request.

[0114] Among them, the heat information is a kind of data object metadata and is stored in the cache pool.

[0115] It should be noted that the data operation request in the above step S24 is any data operation request, such as data operations such as read, write, and update. However, except for the data operation of deletion, there is no need to perform metadata management including heat information on the data after it is deleted. In this embodiment, the metadata of all data objects stored in the distributed storage system are placed in a cache pool composed of storage media with better performance, thereby achieving faster and more efficient metadata management, which is also conducive to improving the reading and writing efficiency of data.

[0116] Furthermore, in addition to flushing cold data in dirty data to the data pool based on the LRU strategy to release space in the cache pool, this embodiment also provides an implementation scheme for migrating data from the data pool to the cache pool to achieve a two-way flow of data between the data pool and the cache pool. Specifically, after the above step S24, the method further includes:

[0117] S25: If the data object is determined to be hot data, a portion of the data in the data object corresponding to the portion stored in the data pool is determined as target data.

[0118] S26: Migrate the target data from the data pool back to the cache pool for storage.

[0119] S27: If the popularity information of the data object has not been updated within a preset period of time, the data object is regarded as dirty data.

[0120] It should be noted that how to determine the data object as hot data in step S25 can be implemented as in the LRU strategy in the above embodiment. That is, the data object that has been recently accessed is determined as hot data based on the LRU strategy. If the data object is in or partially in the data pool, part of the data in the data pool corresponding to the data object is migrated back to the cache pool, so that when a subsequent user accesses it, a faster response can be achieved through the cache pool with higher read and write performance.

[0121] Furthermore, the above embodiments mainly involve the data writing process, and this embodiment also provides a corresponding embodiment for the data reading process, such as Figure 5 As shown, when receiving a data read request sent by the client, the method further includes:

[0122] S41: Determine whether there is data to be read in the buffer pool.

[0123] S42: If it exists, read the partial data matching the data to be read from the cache pool as the first partial data.

[0124] S43: If the first part of data includes all the data to be read, the first part of data is returned as a response result.

[0125] S44: If the first part of data does not include all the data to be read, other data in the data to be read that is not covered by the first part of data is used as the second part of data.

[0126] S45: Read the second part of data from the data pool, and return the first part of data and the second part of data as a response result.

[0127] S46: Write the second portion of data into the cache pool.

[0128] As can be seen from the above, steps S41~S45 are the processing flow of a data read request corresponding to a data reading method provided by the present application. When part of the data to be read is cached in the data pool, it is necessary to read this part of the data (i.e., the second part of the data) from the data pool. And when it is returned as a response result, it can be seen from the heat update scheme of the above embodiment that the heat information of all data objects involved in the data to be read is updated, and since it has just been read, it can be determined as hot data based on the LRU strategy. It can be seen from the hot data migration scheme provided by the above embodiment that the second part of the data in the data pool needs to be migrated back to the cache pool. In the read processing flow provided by the present embodiment, after the cache pool master OSD reads the second part of the data from the data pool and returns it to the client as a response result, it can directly write the second part of the data into the cache pool for later use, rather than the original practice of directly deleting the data in the cache pool after reading the data in the data pool and returning it as a response result.

[0129] That is, this method uses the cache pool as a transfer between the data pool and the client, and the client needs to transfer the data in the data pool through the cache pool. Then, the data being read is necessarily hot data based on the LRU strategy. After the data in the data pool is read into the cache pool, this data is not only returned as a response result, but also written into the cache pool as hot data to bring higher access efficiency for subsequent client access.

[0130] In addition, it should be noted that the second part of the data is not treated as dirty data for a period of time after being written into the cache pool based on this embodiment. The reason is that the dirty data flushing mechanism provided in the above embodiment - the stripe-aligned part of the dirty data and the data to be written is directly flushed to the data pool. Since the first part of the data is also stored in the cache pool, and the collection of the first part of the data and the second part of the data must be stripe-aligned, if the second part of the data is treated as dirty data, it will be flushed to the data pool at the next data write after being written to the cache pool (because the dirty data includes both the first part of the data and the second part of the data, it will be detected in the stripe alignment detection in step S11). Therefore, the above S27 will treat the migrated data as dirty data for subsequent flushing management only after it has not been accessed for a period of time, to ensure that the migrated hot data will not be flushed too quickly, resulting in invalid migration.

[0131] On the other hand, when flushing dirty data, this embodiment also provides another possible implementation scheme. After each flush of dirty data is completed, the method further includes:

[0132] S51: Obtain the total time consumed to complete the flushing of dirty data.

[0133] S52: If the total time consumption exceeds the preset time consumption threshold, the refresh permission flag position is set to a pause state.

[0134] The down-brush permission flag includes: an enable state and a pause state. The down-brush permission flag is in the enable state by default. When the down-brush permission flag is in the enable state, dirty data down-brush is allowed. When the down-brush permission flag is in the pause state, dirty data down-brush is prohibited.

[0135] That is, in this embodiment, after each dirty data flush is completed, the total time consumed to write the dirty data to the data pool is also checked. If the write pressure of the data pool is normal, the total time consumed to complete the dirty data write to the data pool is small and within a preset range (i.e., less than the time consumption threshold). However, if the total time consumed to complete the dirty data write to the data pool exceeds the time consumption threshold, it means that the write pressure of the data pool is too high at this time, and the write performance bottleneck may be reached or is about to be reached. At this time, this embodiment suspends the background dirty data flush to avoid causing greater write pressure on the data pool and affecting the normal write function of the data pool.

[0136] In addition to the embodiment of a data storage method provided in the above embodiment, the present invention also provides an embodiment corresponding to a computer program product. A computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the data storage method described in any of the above embodiments can be implemented.

[0137] Since the embodiments of the computer program product part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the computer program product part, and will not be repeated here.

[0138] In the above embodiment, a data storage method is described in detail, and the present invention also provides an embodiment of a data storage device. It should be noted that the present invention describes the embodiment of the device part from two perspectives, one is based on the functional module perspective, and the other is based on the hardware perspective.

[0139] Based on the functional module perspective, such as Figure 6 As shown, this embodiment provides a data storage device, which is applied to a distributed storage system. The distributed storage system includes a cache pool and a data pool. The device includes:

[0140] The checking module 11 is used to perform a stripe alignment check on the data to be written and the dirty data stored in the current cache pool as a whole when receiving the data to be written, so as to determine the stripe-aligned part of the data to be written and the dirty data, and the non-stripe-aligned part of the data to be written.

[0141] The flush module 12 is used to write the stripe-aligned part of the data to be written and the dirty data into the data pool for storage.

[0142] The cache module 13 is used to write the non-stripe-aligned part of the data to be written as new dirty data into the cache pool for storage.

[0143] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0144] Figure 7 A structural diagram of a data storage device provided in another embodiment of the present invention is shown as follows: Figure 7 As shown, a data storage device comprises: a memory 20 for storing a computer program;

[0145] The processor 21 is used to implement the steps of a data storage method in the above embodiment when executing a computer program.

[0146] The data storage device provided in this embodiment may include but is not limited to a mobile terminal, a personal computer, a workstation, etc.

[0147] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0148] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of a data storage method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. Data 203 may include but is not limited to a data storage method, etc.

[0149] In some embodiments, a data storage device may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0150] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation of a data storage device, and may include more or less components than shown in the figure.

[0151] A data storage device provided by an embodiment of the present invention includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method: a data storage method.

[0152] Finally, the present invention also provides an embodiment corresponding to a non-volatile storage medium. The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.

[0153] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.

[0154] The above is a detailed introduction to a data storage method, device and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0155] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A data storage method, characterized in that: Applied to a distributed storage system, the distributed storage system includes a cache pool and a data pool, and the method includes: When receiving the data to be written, performing a stripe alignment check on the data to be written and the dirty data currently stored in the buffer pool as a whole to determine the stripe-aligned portion of the data to be written and the dirty data, and the non-stripe-aligned portion of the data to be written; Writing the data to be written and a portion of the dirty data whose stripes are aligned into the data pool for storage; Part of the data to be written that is not stripe-aligned is written into the buffer pool as new dirty data for storage.

2. The data storage method according to claim 1, characterized in that: Also includes: Divide the dirty data into stripes, and use stripe data belonging to the same stripe as a group of data objects to be flushed; According to the different data volumes, each of the data objects to be flushed is added to the corresponding least recently used management queue; wherein different least recently used management queues are used to manage the data objects to be flushed with different data volumes, and each least recently used management queue manages the data objects to be flushed in the queue with the least recently used strategy; When the dirty data needs to be flushed to the data pool, the data objects to be flushed are preferentially selected from the least recently used management queue with a large management data volume for flushing.

3. The data storage method according to claim 2, characterized in that: Each time after the dirty data is flushed, the method further includes: Get the current water level of the cache pool; Determine the corresponding refresh cycle according to the water level interval to which the current water level belongs; wherein the water level interval is a plurality of intervals pre-divided for the water level of the buffer pool; and the higher the water level corresponding to the water level interval, the shorter the corresponding refresh cycle; After waiting for the time corresponding to the refresh cycle, the next dirty data refresh action is triggered.

4. The data storage method according to claim 3, characterized in that: The water level of the cache pool is divided into a plurality of water level intervals according to a plurality of pre-set water level nodes, and each of the water level nodes uniquely corresponds to a level down refresh cycle; Then, determining the corresponding next refresh cycle according to the water level interval to which the current water level belongs includes: Determine an intermediate coefficient according to two groups of water level nodes corresponding to the upper limit water level value and the lower limit water level value of the water level interval to which the current water level belongs and the refresh delay; Determine an intermediate result according to the current water level and the first coefficient; The refresh cycle corresponding to the current water level is determined according to the intermediate result and the refresh delay corresponding to the lower limit water level value of the water level interval to which the current water level belongs.

5. The data storage method according to claim 2, characterized in that: Whenever a data operation request is received, the method further comprises: Updating the popularity information of the data object targeted by the data operation request; The heat information is a type of data object metadata and is stored in the cache pool.

6. The data storage method according to claim 5, characterized in that: After updating the heat information of the data object targeted by the data operation request, the method further includes: If the data object is determined to be hot data, a portion of the data in the data object corresponding to the portion stored in the data pool is determined as target data; Migrating the target data from the data pool back to the cache pool for storage; If the heat information of the data object is not updated within a preset period of time, the data object is regarded as the dirty data.

7. The data storage method according to any one of claims 1 to 6, characterized in that: Each time after the dirty data is flushed, the method further includes: Get the total time required to complete the flushing of the dirty data. If the total time consumption exceeds the preset time consumption threshold, the position of the down-sweep permission mark is set to a pause state; Among them, the down-brush enable flag includes: an enable state and a pause state, and the down-brush enable flag defaults to the enable state; when the down-brush enable flag is in the enable state, the down-brush of the dirty data is allowed, and when the down-brush enable flag is in the pause state, the down-brush of the dirty data is prohibited.

8. A data storage device, characterized in that: Applied to a distributed storage system, the distributed storage system includes a cache pool and a data pool, and the device includes: A checking module, for, when receiving data to be written, performing a stripe alignment check on the data to be written and the dirty data currently stored in the cache pool as a whole, so as to determine the stripe-aligned portion of the data to be written and the dirty data, and the non-stripe-aligned portion of the data to be written; A flush module, used for writing the data to be written and the stripe-aligned part of the dirty data into the data pool for storage; The cache module is used to write the non-stripe-aligned part of the data to be written as new dirty data into the cache pool for storage.

9. A data storage device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data storage method according to any one of claims 1 to 7 when executing the computer program.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data storage method according to any one of claims 1 to 7 are implemented.

Images