A distributed all-flash storage data writing method, device and medium

Abstract

The present invention discloses a distributed all-flash storage data writing method, device and medium, which relate to the field of distributed storage technology and are used to optimize the writing performance of all-flash storage. In view of the problem that the current aggregated writing scheme still has room for further improvement in optimizing the writing performance, a distributed all-flash storage data writing method is provided. This method reuses EC data blocks as replica data blocks. Not only can it still provide the optimized writing purpose of aggregating small block IO into large block IO writing, but it also does not require the establishment of additional replica cache partitions, thereby avoiding additional replica management costs. In addition, when the business data is persistently stored in the form of EC, this method also reduces the repeated writing of correction and erasure data, thereby reducing data writing overhead, which is beneficial to improving the storage performance of the distributed all-flash storage system.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a distributed all-flash storage data writing method, device and medium. Background Art

[0002] In related technologies, to improve the write performance of distributed all-flash storage systems in erasure coding (EC) scenarios, a method is proposed to aggregate random small-block input / output (IO) into large-block IO through three replicas in an append-write manner, and then asynchronously flush the large-block IO to the backend storage in the form of EC for persistent storage.

[0003] However, the above solution requires a replica cache as a transit point. When a small I / O block is written, it is first written to the replica cache. When the number of data chunks in the replica cache reaches a certain threshold (i.e., it is combined into a large I / O block), the data in the replica cache is asynchronously flushed to the backend storage in the form of EC for persistent storage. After the flush is complete, the replica cache is notified to delete the data.

[0004] This requires adding a replica cache, which means adding physical storage media at the physical layer as a hardware carrier for the replica cache, or creating additional storage partitions, which in turn increases management costs. Furthermore, the existence of the replica cache and the process of flushing data to EC format for persistent storage behind the replica cache introduces write amplification, negatively impacting storage performance.

[0005] Therefore, technicians in this field are in urgent need of a distributed all-flash storage data writing method to solve the additional copy management costs and write amplification problems brought about by traditional write optimization solutions. Summary of the Invention

[0006] The purpose of the present invention is to provide a distributed all-flash storage data writing method, device and medium for solving the additional copy management cost and write amplification problems brought about by traditional write optimization solutions.

[0007] To solve the above technical problems, the present invention provides a distributed all-flash storage data writing method, comprising:

[0008] When receiving business data, the business data is written into the replica cache data block as replica data; wherein the replica cache data block is N data blocks in the erasure group; wherein N is a positive integer;

[0009] When the amount of the replica data reaches a preset threshold, performing erasure calculation on the replica data to obtain K copies of erasure data and M copies of check data; wherein K and M are both positive integers, and K ≥ N;

[0010] Mapping N of the K copies of the erasure data one by one to corresponding portions of the replica data in the N replica cache data blocks;

[0011] The remaining erasure data and the check data are written into a data block in the erasure group that does not include the duplicate data.

[0012] In a possible embodiment, after writing the remaining erasure data and the check data into the data block in the erasure group that does not include the duplicate data, the method further includes:

[0013] Deleting redundant parts of the replica data in the N replica cache data blocks;

[0014] The redundant part is data in the copy data that has no mapping relationship with the erasure data.

[0015] In a possible embodiment, when writing the service data into the Xth copy cache data block, X is a positive integer, X≤N; the method further includes:

[0016] If the remaining space of the Xth replica cache data block is less than the preset threshold and greater than the total data size of the K copies of the erasure data, then write the business data to the Xth replica cache data block until the remaining space of the Xth replica cache data block is less than the total data size of the K copies of the erasure data;

[0017] The X+1th data block in the erasure group is used as the replica cache data block and the business data is written into it.

[0018] In a possible embodiment, after deleting the redundant parts of the replica data in the N replica cache data blocks, the method further includes:

[0019] If the remaining space of the Xth replica cache data block is greater than the total size of the K copies of the erasure data, writing the portion of the replica data in the X+1th data block of the erasure group that is mapped to the erasure data into the Xth replica cache data block;

[0020] A portion of the replica data in the X+1th data block of the erasure correction group that is mapped to the erasure correction data is deleted.

[0021] In a possible embodiment, when writing the service data into the replica cache data block as replica data, the method further includes:

[0022] Writing the business data and the mapping relationship between the business data and the replica cache data block into the cache;

[0023] Then, when the amount of the replica data reaches a preset threshold, the step of performing erasure calculation on the replica data to obtain K pieces of erasure data and M pieces of check data includes:

[0024] Erasure calculation is performed using the business data in the cache.

[0025] In a possible embodiment, after receiving the service data, the method further includes:

[0026] Determine whether there is a data block in the open state;

[0027] If not, apply to the metadata management server for Y data blocks as the erasure group; wherein Y is a positive integer and Y ≥ K + M;

[0028] M data blocks in the erasure group are selected as the replica cache data blocks; wherein the replica cache data blocks are preferentially selected from the first K data blocks in the erasure group.

[0029] In a possible embodiment, the method further includes:

[0030] After the business data is written into the replica cache data block, updating metadata information of the business data and metadata information of the data block;

[0031] The metadata information of the business data includes: data block number, offset, data length and version number;

[0032] The metadata information of the data block includes: space bitmap information, object list and data block version number;

[0033] After mapping the erasure data to a corresponding portion of the replica data in the replica cache data block, updating the data block version number of the replica cache data block;

[0034] After writing the remaining erasure data or the check data into the data block of the erasure group, updating metadata information of the data block;

[0035] After deleting the data portion of the replica data in the replica cache data block that does not establish a mapping relationship with the erasure data, the space bitmap information of the replica data block is updated.

[0036] To solve the above technical problems, the present invention further provides a distributed all-flash storage data writing device, comprising:

[0037] A replica cache module, configured to write service data into a replica cache data block as replica data when receiving the service data; wherein the replica cache data block is N data blocks in an erasure group; wherein N is a positive integer;

[0038] an erasure calculation module, configured to perform erasure calculation on the replica data to obtain K copies of erasure data and M copies of check data when the amount of the replica data reaches a preset threshold; wherein K and M are both positive integers, and K ≥ N;

[0039] an erasure mapping module, configured to map N of the K erasure data to corresponding portions of the replica data in the N replica cache data blocks;

[0040] An erasure writing module is configured to write the remaining erasure data and the check data into a data block in the erasure group that does not include the duplicate data.

[0041] To solve the above technical problems, the present invention further provides a distributed all-flash storage data writing device, comprising:

[0042] memory for storing computer programs;

[0043] The processor is configured to implement the steps of the distributed all-flash storage data writing method described above when executing the computer program.

[0044] To solve the above technical problems, the present invention further provides a non-volatile storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the distributed all-flash storage data writing method described above are implemented.

[0045] The present invention provides a distributed all-flash storage data writing method that reuses EC data blocks as replica cache data blocks. During the process of aggregating multiple small I / O blocks into a large I / O write, the small I / O blocks (i.e., business data) are written to the replica cache data blocks. In this method, the replica cache data blocks do not require additional partitioning, but instead reuse existing data blocks in the EC group. When the amount of written replica data reaches a preset threshold, erasure coding is performed on the replica data for persistent storage in EC format. In this method, the replica cache data blocks are data blocks in the EC group, meaning that N data blocks in the EC group contain all business data, i.e., these N data blocks contain all EC data. Therefore, when writing K copies of erasure data, N of these copies do not need to be written repeatedly; instead, a mapping relationship needs to be established with the corresponding data portions in the replica data. Subsequently, only the remaining erasure data (possibly none, if N equals K) and M copies of parity data need to be written to the EC group data blocks for persistent storage.

[0046] As can be seen from the above, this method reuses EC data blocks as replica data blocks. This not only provides the optimized write performance of aggregating small I / O blocks into large I / O blocks, but also eliminates the need to establish additional replica cache partitions, thus avoiding additional replica management costs. Furthermore, when business data is persistently stored in EC format, this method reduces the duplication of erasure data, thereby reducing data write overhead and improving the storage performance of distributed all-flash storage systems.

[0047] The distributed all-flash storage data writing device and non-volatile storage medium provided by the present invention correspond to the above method and have the same effects as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 This is a flowchart of an IO aggregation writing method;

[0050] Figure 2 A schematic diagram of writing at the physical medium layer using an IO aggregation writing method;

[0051] Figure 3 A flowchart of a distributed all-flash storage data writing method provided by an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of writing data at the physical medium layer of a distributed all-flash storage data writing method provided by an embodiment of the present invention;

[0053] Figure 5 A schematic diagram of a redundant copy data clearing solution provided by an embodiment of the present invention;

[0054] Figure 6 A schematic diagram of data transfer of a data block space utilization solution provided by an embodiment of the present invention;

[0055] Figure 7 A schematic diagram of a data mapping write cache solution provided by an embodiment of the present invention;

[0056] Figure 8 This is a schematic diagram of the management process of an IO aggregation writing method;

[0057] Figure 9 A structural diagram of a distributed all-flash storage data writing device provided by an embodiment of the present invention;

[0058] Figure 10 This is a structural diagram of another distributed all-flash storage data writing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0060] The core of the present invention is to provide a distributed all-flash storage data writing method, device and medium.

[0061] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0062] All-flash storage, or a storage system built entirely from flash storage media (such as NVMe SSDs), can achieve sub-millisecond latency and extremely high random read and write performance. This, combined with a distributed storage system design, results in a distributed all-flash storage system capable of parallel processing requests across multiple nodes, delivering even higher storage performance.

[0063] In distributed all-flash storage, small random writes in erasure coding (EC) scenarios require first reading data across nodes, then writing it to the solid-state disk (SSD) after the modification is complete. This process not only incurs additional read overhead, but small-block writes and erases also impact the SSD's garbage collection (GC) efficiency and write amplification. Therefore, write performance in erasure coding scenarios has become a key optimization goal for all-flash storage.

[0064] In related technologies, such as Figure 1 and Figure 2 As shown in the figure, a solution is proposed to aggregate random small-block input / output (IO) into large-block IO through three replicas in an append-write manner, and then asynchronously flush the large-block IO to the backend storage in the form of EC for persistent storage.

[0065] in, Figure 1The storage resources provided by the actual physical storage media are abstracted to create a replica cache and an EC persistent storage area. For data sent from the protocol service layer, large-block I / O is directly written to the EC persistent storage area using erasure coding. Small-block I / O is first written to the replica cache (using three replicas as an example, this means three replicas of data are retained in the replica cache) until the replica size reaches the upper limit (e.g., 4MB). At this point, the aggregated business data is written to the EC persistent storage area using an append write to complete the data's persistent storage. Finally, the replica cache is notified to delete the cached data, completing the aggregate write.

[0066] In the above process, the replica cache is user-unaware and operates entirely in an internal append-write mode. Writing objects involves replica management, object application, relationship mapping, and object release and recovery. Object metadata is stored after the object data is written to the replica cache. After multiple cached copies of data reach a certain threshold, they are flushed to the EC. This process also involves EC unit management, object application, relationship mapping, and garbage collection. After the EC is flushed, the mapping between objects and data chunks is updated, and the cached replica chunks are released. This solution, however, suffers from write amplification and offers room for further optimization.

[0067] Figure 2 corresponds to Figure 1 The process flow at the physical media layer is schematically illustrated. The aforementioned replica cache area is also implemented using storage resources provided by the physical layer. This is achieved by demarcating a storage area within the physical layer specifically for replica caching. However, due to replica caching requirements, the physical media used for the replica cache area must utilize high-performance storage devices such as Storage Class Memory (SCM) or Single Level Cell (SLC). Due to the varying storage requirements of different data, the physical media layer typically requires multiple storage media to implement EC persistent storage and replica caching, increasing the complexity and implementation difficulty of the physical media layer. Furthermore, adding partitions such as SCM / SLC to the physical media layer increases management costs for replicas.

[0068] Based on the above problems, the present invention provides a distributed all-flash storage data writing method, such as Figure 3 Shown, including:

[0069] S11: When receiving service data, the service data is written into the replica cache data block as replica data.

[0070] The replica cache data blocks are N data blocks in the erasure group, where N is a positive integer.

[0071] S12: When the amount of the replica data reaches a preset threshold, erasure calculation is performed on the replica data to obtain K copies of erasure data and M copies of verification data.

[0072] Wherein, K and M are both positive integers, K≥N.

[0073] It should be noted that erasure coding is a data protection method that segments data into fragments, expands and encodes redundant data blocks, and stores them in different locations. Specifically, erasure coding divides the original data to be protected into K equal parts, resulting in K pieces of erasure-corrected data. Each piece of erasure-corrected data is a portion of the original data (1 / K). Then, the K pieces of erasure-corrected data are encoded and calculated using an erasure coding algorithm to produce M pieces of parity data. The parity data is not part of the original data, but is obtained by performing erasure coding on the original data. The parity data can be restored back to a portion of the original data through erasure decoding (the inverse process of erasure coding). In other words, after the original data is divided into K pieces of erasure-corrected data, as long as the number of missing pieces of erasure data does not exceed M, the M pieces of parity data can be used to restore the complete original data through erasure decoding, thereby achieving data redundancy protection. The erasure ratio K:M in the erasure code directly affects the redundancy level after the erasure coding calculation and can be freely set according to actual needs. This embodiment does not impose any restrictions here.

[0074] S13: Map N of the K copies of erasure data one by one to corresponding parts of the replica data in the N replica cache data blocks.

[0075] It is not difficult to understand that since the replica data is written in only N data blocks in the erasure group, only the replica data in these N data blocks can be mapped into N different erasure data.

[0076] S14: Writing the remaining erasure data and check data into the data blocks in the erasure group that do not contain duplicate data.

[0077] It should be noted that this method is a further solution provided on the basis of the data aggregation writing method provided above. Figure 1 As shown in the figure, the writing process of this method is not significantly different from the above solution at an abstract level. Similarly, small blocks of IO (i.e., business data) are first written to the replica cache. When the data in the replica cache reaches a certain amount, it is written to the backend EC persistent storage in an append-only manner.

[0078] However, it's important to note that this method doesn't require a separate partition in the physical media layer as a replica cache. In this method, the replica cache data blocks used to cache replica data are also the data blocks in the EC group used to implement EC persistent storage in the physical media layer. In other words, the replica data and the subsequently persistently stored EC data can share the same data block.

[0079] The data writing process is as follows Figure 4 As shown, Figure 2 The examples are the same as Figure 4 Here, the number of replicas is 3, and the EC ratio is 4 + 2. That is, N = 3, K = 4, and M = 2. When the protocol service layer issues business data, it is first written to the replica cache data block. The replica cache data block is the data block in the EC pool that will eventually be archived. In other words, the data block containing the replica data is a subset of the data block containing the EC to be flushed and archived. Later, when the replica data accumulates to a certain size (for example, the data volume reaches 4MB, meaning the size of a replica cache data block is 4MB), a background flush is initiated, converting the replica data into EC data (4 copies of erasure data and 2 copies of parity data). According to the principles of erasure code calculation, erasure code calculation divides the target data into K copies of erasure data, then calculates M copies of parity data from these K copies. Therefore, any copy of erasure data is a subset of the replica data. That is, the replica data stored in any replica cache data block necessarily includes any copy of erasure data.

[0080] Based on this, if N is 3 and K is 4 in the above example, then when writing EC data, three of the four copies of the erasure data do not need to be written repeatedly. The replica data in the three replica cache data blocks contains these three copies of the erasure data. In this case, corresponding to step S13 above, there is no need to send a write command with the data content (erasure data). Instead, the portion of the data in the replica cache data block corresponding to the erasure data to be written can be directly mapped as the erasure data. In other words, a mapping relationship is established, specifying a portion of the replica data stored in the replica cache data block as the erasure data to be written, rather than actually writing the erasure data.

[0081] Next, step S14 involves writing the remaining EC data that could not be directly mapped. Since K is 4 and N is 3 in this example, K>N, some erasure data remains that cannot be mapped. (If N ≥ K, no erasure data remains, and all erasure data can be mapped as part of the replica data.) Furthermore, since the check data is calculated based on the target data (i.e., the data for which the erasure code is calculated), it is not a subset of the target data, nor a subset of the replica data. Therefore, it cannot be mapped and must be written independently.

[0082] However, it's easy to understand that this solution eliminates the need to write a certain amount of data. Let's take a scenario with 3 replicas and an EC ratio of 4+2 as an example. Assuming the preset threshold is W, that is, the amount of data in one replica is W. The following compares the amount of data written in various scenarios for this solution and those in related technologies:

[0083] 1. Brush up the bandwidth in the background.

[0084] The writing amount of this method is: W*3 / 4 =0.75W.

[0085] The original solution writes: W*6 / 4 = 1.5W.

[0086] Total write reduction: 50%.

[0087] 2. Aggregate small IO blocks into large IO blocks of size W, and start background flushing to write the data to the total write amount of EC.

[0088] The writing amount of this method is: 3W+W*3 / 4 =3.75W.

[0089] The original solution writes: 3W + W*6 / 4 = 4.5W.

[0090] Total write volume reduction: 16.67%.

[0091] As can be seen, this method can effectively reduce the amount of data written during background flushing, thereby reducing the overall amount of data written when performing small-block IO aggregation writes. Furthermore, as can be seen above, the replica cache data blocks used to cache replica data in this method are the data blocks in the EC pool that will ultimately be archived. In other words, in this method, the system no longer requires a separate replica cache area, only a single EC pool. The physical media layer can be simplified using a single medium, such as Triple-Level Cell (TLC). Furthermore, the lack of independent storage partitions eliminates additional management costs. In summary, compared to the small-block IO aggregation write solution proposed in the related art, this method optimizes data layout by reusing replica chunks as EC chunks. This not only improves small IO random write performance, but also reduces cluster background flushing bandwidth and the total SSD write volume. It also simplifies the operational and management overhead of system replicas and chunks, improving the write performance of distributed all-flash storage systems.

[0092] Furthermore, in related technologies, after the off-stage refresh is completed, the copy data in the copy cache area needs to be cleared. Correspondingly, in this solution, it is necessary to clear the useless copy data. Based on this, this embodiment provides a possible solution that is suitable for this. After step S14, the above method further includes:

[0093] S15: Delete the redundant parts of the replica data in the N replica cache data blocks.

[0094] The redundant part is the data in the replica that is not mapped to the erasure data.

[0095] As can be seen from the above embodiments, this method reduces the repeated writing of erasure data by mapping the duplicate data into erasure data. Figure 5 As shown, for the replica data, the data portion mapped as erasure data does not belong to redundant data after the background flush is completed and needs to be retained, but the other data portions belong to redundant data and need to be cleared.

[0096] Furthermore, the amount of redundant data required to be deleted in this embodiment is also advantageous compared to solutions in related technologies:

[0097] The amount deleted by this method is: 3W *3 / 4 =2.25W.

[0098] Amount deleted from the original plan: 3W.

[0099] Total deletions decreased: 25%.

[0100] As can be seen, this method, by reusing replica data and mapping it as erasure data, not only eliminates the need to repeatedly write this mapped erasure data, but also eliminates the need to delete this portion of data reused as erasure data when deleting redundant replica data after a background refresh. In other words, this method offers the advantage of requiring less data for both data writing and data deletion, thereby improving the overall performance of the distributed all-flash storage system.

[0101] On the other hand, this embodiment also provides a possible implementation scheme. After receiving the service data and before writing the service data into the replica cache data block as the replica data, the above method further includes:

[0102] S16: Determine whether the volume of the business data exceeds a preset threshold. If not, go to step S11; if so, go to step S17.

[0103] S17: Perform erasure code calculation on the business data, and write the obtained K pieces of erasure data and M pieces of check data into the data block of the erasure group.

[0104] That is, in addition to the aggregate write method, this method also provides a direct write method for large-block IOs whose data volume exceeds a preset threshold. It is easy to understand that the purpose of write aggregation in this method is to optimize the write performance of the distributed all-flash storage system. However, write aggregation will inevitably produce certain write amplification problems, that is, after optimization based on this method, write amplification still exists. Therefore, for business data with a large amount of data, if the aggregation method is still used, the significantly increased data write volume is not conducive to improving the write performance, but will affect the write performance of the distributed all-flash storage system. Based on this, this embodiment provides a further implementation plan, when the protocol service layer sends business data, the data volume of the business data is compared with the preset threshold. If it exceeds the preset threshold, it is determined that this business data is a large-block IO, and no aggregation is required to reduce the data write volume. If it does not exceed the preset threshold, it is determined that this business data is a small-block IO, and the aggregation method provided in the above embodiment can be performed to ensure better write performance.

[0105] On the other hand, this embodiment also provides a possible implementation scheme. When writing the service data to the replica cache data block in step S11, the method further includes:

[0106] S21: If the remaining space of the Xth replica cache data block is less than a preset threshold and greater than the total data size of K copies of erasure data, then write business data to the Xth replica cache data block until the remaining space of the Xth replica cache data block is less than the total data size of K copies of erasure data.

[0107] The Xth replica cache data block is also the replica cache data block for the business data written this time. That is, X is a positive integer, X≤N.

[0108] S22: Use the X+1th data block in the erasure group as a replica cache data block and write the business data into it.

[0109] It's easy to understand that this method significantly reduces the amount of data written during background flushing by reusing data blocks in the erasure group as replica cache blocks. However, because the data blocks in the erasure group are also used to persistently store EC data, as business data is continuously flushed, EC data will gradually be stored in the data blocks used as replica cache blocks.

[0110] Therefore, with respect to the use of data blocks, initially there is no data stored in the data blocks, and the above-mentioned preset thresholds are generally smaller than the remaining space of the data blocks. As data is continuously stored in the data blocks, the remaining space of the data blocks begins to gradually decrease. Based on this, the present embodiment provides a method for utilizing the storage space of replica cache data blocks, as long as the replica cache data blocks have remaining space that can store K copies of erasure data. That is, there is no need to apply for a separate data block as a new replica cache data block, maximizing the use of the space of the replica cache data blocks. K copies of erasure data are also one copy of data, which is the size of the preset threshold. In other words, as long as the replica cache data block still has space that can be used to map the replica data once to reuse it as erasure data, it will continue to be used.

[0111] In addition, if this embodiment is also applied in conjunction with step S15 provided in the above embodiment, it can be seen from step S15 provided in the above embodiment that after completing the off-stage refresh, it is necessary to clear the redundant copy data in the replica cache data block. This step will free up a certain amount of space in the replica cache data block. If this step causes the remaining space of the replica cache data block to be greater than the preset threshold again, this replica cache data block will continue to participate in the next business data aggregation until the remaining space of the replica cache data block is no longer greater than the preset threshold.

[0112] Based on the above embodiment, this embodiment further provides a further data block space utilization solution. After step S15, the above method further includes:

[0113] S23: If the remaining space of the Xth replica cache data block is larger than the total size of the K copies of erasure data, write the portion of replica data in the X+1th data block of the erasure group that is mapped to the erasure data into the Xth replica cache data block.

[0114] S24: Delete some replica data in the X+1th data block of the erasure correction group that is mapped to the erasure correction data.

[0115] It is easy to understand that the Xth replica cache data block is the target replica cache data block for a single execution of steps S23 and S24 provided in this embodiment. The X+1th replica cache data block is the next data block after the target replica cache data block, i.e., the data block that needs to be applied for as a new replica cache data block when the remaining space of the Xth replica cache data block is insufficient.

[0116] As can be seen from the above embodiments, the data block space utilization solution provided in the above embodiments can ensure that the remaining space of each replica cache data block does not exceed a preset threshold (e.g., 4MB) when it is no longer in use, thereby effectively utilizing the data block storage space. However, this solution also provides a further space utilization solution based on this:

[0117] Based on the calculation principles of erasure codes, a copy of erasure data is 1 / K of the target data. In this method, the target data is also the replica data, and the size of the replica data is determined by a preset threshold. Assuming the preset threshold is W, the size of the replica data is also W. Continuing with the example above where K is 4, the size of a copy of erasure data is W / 4. It's easy to see that this method ultimately uses EC data for persistent storage of business data. Therefore, even if the remaining space in the replica cache data block is less than W, it is likely to be greater than W / 4.

[0118] Based on this, the solution provided by this embodiment further improves the utilization rate of the replica cache data block space. When the remaining space of the replica cache data block is less than W, it cannot cache the replica data, that is, it cannot map the erasure data. At this time, it is necessary to apply for the next data block in the erasure group as a new replica cache data block (the X+1th) to cache the replica data. However, based on step S15 provided in the above embodiment, it can be seen that after the background refresh is completed, the redundant replica data will be cleared. At this time, what needs to be stored is no longer the replica data with a data volume of W, but the erasure data with a data volume of W / K. Therefore, if Figure 6 As shown in , if the remaining space of the Xth replica cache data block is greater than W / K, the erasure data in the X+1th replica cache data block is transferred to the Xth replica cache data block to make greater use of the storage space of the Xth replica cache data block. It should also be noted that, if Figure 6 As shown, when the erasure data in the X+1th replica cache data block is transferred to the Xth replica cache data block, stripe alignment is automatically performed based on the erasure code algorithm.

[0119] Based on the solution provided by this embodiment, the remaining unused space of each data block can be reduced to no more than W / K, further improving the space utilization of the data blocks. It should also be noted that although this embodiment generates a certain amount of data transfer and writes (W / K), this amount of data writes is very small compared to the overall data write volume of the distributed storage cluster. Furthermore, the frequency of occurrence is not high, with a replica cache data block experiencing at most three transfers. The additional available space created by this method is more significant. Therefore, the small amount of data transfer introduced by this method is acceptable compared to the resulting effect, and can even be ignored.

[0120] On the other hand, this embodiment also provides a possible implementation scheme. When the business data is written into the replica cache data block as replica data in step S11, the method further includes:

[0121] S18: Write the business data and the mapping relationship between the business data and the replica cache data block into the cache.

[0122] Then step S12 specifically includes: performing erasure calculation using the business data in the cache.

[0123] That is, Figure 7 As shown, when the present embodiment completes writing the business data into the replica cache data block as replica data, it also writes a replica mapping into the cache of the distributed all-flash storage system. So that when the background flush is performed later, there is no need to re-query the corresponding replica data from the physical medium layer and read it to perform erasure code calculation and complete the backstage flush. Through the mapping saved in the cache, the location of the replica data can be quickly determined, thereby improving the read hit and flush processing performance. In addition, the present method only needs to save a replica mapping in the cache. After the backstage flush is completed, the mapping can be deleted from the cache. The cache space occupied can be ignored and will not affect the performance of the distributed all-flash storage system.

[0124] On the other hand, this embodiment also provides a possible implementation scheme. After receiving the service data, the method further includes:

[0125] S31: Determine whether there is a data block in the open state. If not, go to step S32.

[0126] S32: Request Y data blocks from the metadata management server as an erasure group.

[0127] Wherein, Y is a positive integer, and Y≥K+M.

[0128] S33: Select M data blocks in the erasure group as replica cache data blocks.

[0129] Among them, the replica cache data blocks are preferentially selected from the first K data blocks in the erasure group.

[0130] That is, this embodiment provides a data block usage solution. Specifically, when business data is written (specifically, it can be written in an append-write manner through the replica interface layer), the replica cache area determines whether there is a data block in the open (OPEN) state, that is, whether there is an available replica cache data block. If not, the metadata management server (Metaserver) is requested for data blocks to form an erasure group, and M data blocks are selected as a virtual replica group, that is, physical storage resources are provided for the replica cache area. Among them, the first M data blocks in the erasure group are preferentially selected as replica cache data blocks to adapt to the habit of storing business data in the form of EC in the physical medium layer, that is, the replica cache data blocks can be reused as EC data blocks that store erasure data, thereby improving storage efficiency.

[0131] On the other hand, Figure 8As shown in the figure, the IO aggregation writing solution provided in the related technology also involves management needs such as EC unit management, object application, relationship mapping, GC processing, etc. After the background flush is completed, it is necessary to update the mapping relationship between the business data object (Object) and the data block, and release the data block in the copy cache. The management process involved is as follows Figure 8 As shown by arrows 1 to 5 in FIG, this is achieved through the metadata management server.

[0132] Based on this, this embodiment also provides a possible management solution, and the above method further includes:

[0133] S41: After the service data is written into the replica cache data block, metadata information of the service data and metadata information of the data block are updated.

[0134] The metadata information of business data includes: data block number (chunk id), offset (offset), data length (len) and version number (version).

[0135] The metadata information of a data block includes: space bitmap information (bitmap), object list and data block version number.

[0136] S42: After mapping the erasure correction data to the corresponding portion of the replica data in the replica cache data block, updating the data block version number of the replica cache data block.

[0137] S43: After writing the remaining erasure data or check data into the data block of the erasure group, update the metadata information of the data block.

[0138] S44: After deleting the data portion of the replica data in the replica cache data block that does not establish a mapping relationship with the erasure correction data, update the space bitmap information of the replica data block.

[0139] That is, this embodiment provides a metadata management solution for business data objects and data blocks. It should be noted that, because this method reuses replica data as erasure data, the metadata management required by this method is also simpler than the original solution.

[0140] In the original solution, object mapping required metadata such as chunk ID, version, offset, and len. The metadata for the data chunk used as the copy buffer needed to be deleted, and the new data chunk's internal bitmap, used / free amount, and object identifier (OID) needed to be updated.

[0141] In this method, all other aspects are the same, but this method reuses EC data blocks as replica cache data blocks. Therefore, when clearing replica data, only the redundant data portion needs to be cleared, so there is no need to delete the metadata of the replica cache data block, reducing one metadata management operation. In summary, the distributed all-flash storage data writing method provided by the present invention has greater advantages in background write volume, total write volume, and replica management, and the lack of need to set up an independent replica buffer is conducive to simplifying the physical medium layer. It is superior to the original solution in all aspects.

[0142] In addition to the embodiments of the distributed all-flash storage data writing method provided in the above embodiments, the present invention also provides corresponding embodiments of a computer program product. The computer program product includes a computer program / instructions that, when executed by a processor, implements the steps of the distributed all-flash storage data writing method described in any of the above embodiments.

[0143] Since the embodiments of the computer program product part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the computer program product part, and will not be repeated here.

[0144] In the above embodiment, a distributed all-flash storage data writing method is described in detail. The present invention also provides a corresponding embodiment of a distributed all-flash storage data writing device. It should be noted that the present invention describes the device embodiment from two perspectives: one from the perspective of functional modules and the other from the perspective of hardware.

[0145] Based on the perspective of functional modules, such as Figure 9 As shown, this embodiment provides a distributed all-flash storage data writing device, including:

[0146] The replica cache module 11 is used to write the business data into the replica cache data block as replica data when receiving the business data; wherein the replica cache data block is N data blocks in the erasure group; wherein N is a positive integer.

[0147] The erasure calculation module 12 is used to perform erasure calculation on the replica data to obtain K copies of erasure data and M copies of verification data when the amount of the replica data reaches a preset threshold; wherein K and M are both positive integers, K≥N.

[0148] The erasure mapping module 13 is configured to map N of the K erasure data to corresponding portions of the replica data in the N replica cache data blocks.

[0149] The erasure writing module 14 is configured to write the remaining erasure data and check data into data blocks in the erasure group that do not contain duplicate data.

[0150] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, and they will not be repeated here.

[0151] Figure 10 A structural diagram of a distributed all-flash storage data writing device provided in another embodiment of the present invention is shown as follows: Figure 10 As shown, a distributed all-flash storage data writing device includes: a memory 20 for storing computer programs;

[0152] The processor 21 is configured to implement the steps of a distributed all-flash storage data writing method according to the above embodiment when executing a computer program.

[0153] The distributed all-flash storage data writing device provided in this embodiment may include but is not limited to a mobile terminal, a personal computer, a workstation, and the like.

[0154] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), and a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0155] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of a distributed all-flash storage data writing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include but is not limited to a distributed all-flash storage data writing method, etc.

[0156] In some embodiments, a distributed all-flash storage data writing device may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0157] Those skilled in the art will understand that Figure 10 The structure shown in the figure does not constitute a limitation on a distributed all-flash storage data writing device, and may include more or fewer components than shown in the figure.

[0158] An embodiment of the present invention provides a distributed all-flash storage data writing device, comprising a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method: a distributed all-flash storage data writing method.

[0159] Finally, the present invention also provides an embodiment corresponding to a non-volatile storage medium. The non-volatile storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiment.

[0160] It is understood that if the methods in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0161] The above is a detailed introduction to a distributed all-flash storage data writing method, device and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0162] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. A distributed all-flash storage data writing method, characterized in that: include: When receiving business data, the business data is written into the replica cache data block as replica data; wherein the replica cache data block is N data blocks in the erasure group; wherein N is a positive integer; When the amount of the replica data reaches a preset threshold, performing erasure calculation on the replica data to obtain K copies of erasure data and M copies of check data; wherein K and M are both positive integers, and K ≥ N; Mapping N of the K copies of the erasure data one by one to corresponding portions of the replica data in the N replica cache data blocks; The remaining erasure data and the check data are written into a data block in the erasure group that does not include the duplicate data.

2. The distributed all-flash storage data writing method according to claim 1, characterized in that: After writing the remaining erasure data and the check data into the data block in the erasure group that does not include the duplicate data, the method further includes: Deleting redundant parts of the replica data in the N replica cache data blocks; The redundant part is data in the copy data that has no mapping relationship with the erasure data.

3. The distributed all-flash storage data writing method according to claim 2, characterized in that: When writing the service data into the Xth copy cache data block, X is a positive integer, X≤N; the method further includes: If the remaining space of the Xth replica cache data block is less than the preset threshold and greater than the total data size of the K copies of the erasure data, then write the business data to the Xth replica cache data block until the remaining space of the Xth replica cache data block is less than the total data size of the K copies of the erasure data; The X+1th data block in the erasure group is used as the replica cache data block and the business data is written into it.

4. The distributed all-flash storage data writing method according to claim 3, characterized in that: After deleting the redundant parts of the replica data in the N replica cache data blocks, the method further includes: If the remaining space of the Xth replica cache data block is greater than the total size of the K copies of the erasure data, writing the portion of the replica data in the X+1th data block of the erasure group that is mapped to the erasure data into the Xth replica cache data block; A portion of the replica data in the X+1th data block of the erasure correction group that is mapped to the erasure correction data is deleted.

5. The distributed all-flash storage data writing method according to claim 1, characterized in that: When writing the service data into the replica cache data block as replica data, the method further includes: Writing the business data and the mapping relationship between the business data and the replica cache data block into the cache; Then, when the amount of the replica data reaches a preset threshold, the step of performing erasure calculation on the replica data to obtain K pieces of erasure data and M pieces of check data includes: Erasure calculation is performed using the business data in the cache.

6. The distributed all-flash storage data writing method according to claim 1, characterized in that: After receiving the service data, the method further includes: Determine whether there is a data block in the open state; If not, apply to the metadata management server for Y data blocks as the erasure group; wherein Y is a positive integer and Y ≥ K + M; M data blocks in the erasure group are selected as the replica cache data blocks; wherein the replica cache data blocks are preferentially selected from the first K data blocks in the erasure group.

7. The distributed all-flash storage data writing method according to any one of claims 1 to 6, characterized in that: The method further comprises: After the business data is written into the replica cache data block, updating metadata information of the business data and metadata information of the data block; The metadata information of the business data includes: data block number, offset, data length and version number; The metadata information of the data block includes: space bitmap information, object list and data block version number; After mapping the erasure data to a corresponding portion of the replica data in the replica cache data block, updating the data block version number of the replica cache data block; After writing the remaining erasure data or the check data into the data block of the erasure group, updating metadata information of the data block; After deleting the data portion of the replica data in the replica cache data block that does not establish a mapping relationship with the erasure data, the space bitmap information of the replica data block is updated.

8. A distributed all-flash storage data writing device, characterized in that: include: A replica cache module, configured to write service data into a replica cache data block as replica data when receiving the service data; wherein the replica cache data block is N data blocks in an erasure group; wherein N is a positive integer; an erasure calculation module, configured to perform erasure calculation on the replica data to obtain K copies of erasure data and M copies of check data when the amount of the replica data reaches a preset threshold; wherein K and M are both positive integers, and K ≥ N; an erasure mapping module, configured to map N of the K erasure data to corresponding portions of the replica data in the N replica cache data blocks; An erasure writing module is configured to write the remaining erasure data and the check data into a data block in the erasure group that does not include the duplicate data.

9. A distributed all-flash storage data writing device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the distributed all-flash storage data writing method according to any one of claims 1 to 7 when executing the computer program.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, which, when executed by a processor, implements the steps of the distributed all-flash storage data writing method according to any one of claims 1 to 7.

Images