All-flash array based on software and hardware collaborative design

Abstract

The invention belongs to the technical field of data storage, and relates to an all-flash array based on software and hardware collaborative design. The all-flash array comprises an AFA address space which is used for sequentially recording data written by a user; k continuous data blocks in the AFA address space form a stripe, and each stripe records the storage positions of the k data blocks and the corresponding m verification blocks in the SSD; the storage space of the SSD in the all-flash array is divided into a normal area and an instantaneous area, and the two areas are respectively organized into a horizontal stripe and a vertical stripe; and completing a copying stage and a conversion stage by using the horizontal stripe and the vertical stripe. Through software and hardware collaborative design, the conversion stage is unloaded into the SSD to be completed, so that the problem of performance reduction caused by a large number of I / O between a host and the SSD in the stage is solved; the influence of SSD garbage collection on the array performance is relieved through software and hardware collaborative design, garbage collection detection and garbage collection avoidance.

Description

Technical Field

[0001] The present invention belongs to the technical field of data storage, and particularly relates to an all-flash array based on a software-hardware co-design. Background Art

[0002] A solid state drive (SSD) refers to a hard disk made of solid state electronic storage chips (such as NAND flash memory). An all-flash array (AFA) refers to a storage array composed of multiple SSDs. NAND flash memory does not support in-place overwrite. To hide this physical property from users, when users need to overwrite, the SSD firmware writes the data to a new storage location and marks the original storage location as invalid, which is called garbage collection (GC). However, this will reduce the available space. To solve this problem, the garbage collection process aggregates the valid data and releases the invalid space again.

[0003] All-flash arrays can be divided into two types according to the I / O path: strip write type and two-stage write type. The present invention mainly relates to the two-stage write type all-flash array. Figure 1 Shows the process of two-stage writing. Specifically, the two-stage write type all-flash array writes data blocks in two stages: the copy stage and the conversion stage. In the copy stage, the data blocks are copied m + 1 times and distributed in m + 1 SSDs (①), which provides the same fault tolerance as an all-flash array of strip write type composed of k + m SSDs. When the size of the copied data exceeds a given threshold (for example, 5% of the total capacity of the array), the user I / O request will be suspended and enter the conversion stage. Specifically, the host reads all the data blocks of the same strip from the SSDs (②) and calculates the parity block (③). Then, these data blocks and the parity block are written back to the SSDs and stored in a strip layout (④). Finally, the space previously used to store the copied data blocks will be recycled.

[0004] The disadvantages of the prior art are as follows:

[0005] 1) A large amount of SSD I / O is generated in the conversion stage to read the data into the host, calculate the parity, and then write it back to the SSD, which will cause a significant drop in the performance of the all-flash array.

[0006] 2) Triggering the garbage collection mechanism of the SSD will reduce its performance. And since the all-flash array is composed of multiple SSDs and the I / O operations require multiple SSDs to complete together, after any SSD starts garbage collection, the overall performance of the all-flash array will also drop sharply. Summary of the Invention

[0007] In view of the above problems, the present invention provides an all-flash array based on a software-hardware co-design.

[0008] The technical solution adopted by the present invention is as follows:

[0009] A full flash array based on hardware-software co-design, the full flash array includes an AFA address space for sequentially recording data written by users; consecutive k data blocks in the AFA address space form a stripe, and each stripe records the storage locations of k data blocks and corresponding m parity blocks in the SSD; the storage space of the SSD in the full flash array is divided into a normal area and an instantaneous area, the normal area is used to store long-term existing data, and the instantaneous area is used to store instantaneous data; in the normal area, k+m data blocks and parity blocks with the same SSD logical address from k+m SSDs are combined into a horizontal stripe; in the instantaneous area, consecutive k data blocks in the same SSD form a vertical stripe; a horizontal stripe and m vertical stripes from m different SSDs are combined into a stripe group to track the storage locations of data blocks and parity blocks through the stripe group.

[0010] Further, the full flash array writes data blocks in two stages: a replication stage and a conversion stage.

[0011] Further, the replication stage includes: when writing a data block, the data block is replicated m+1 times, one of which is called the main copy and is stored in the horizontal stripe, and the other m copies are called slave copies and are written into m vertical stripes.

[0012] Further, the conversion stage includes:

[0013] When the host is idle or the vertical stripe is insufficient, the conversion stage starts, and the conversion task is offloaded to the SSD;

[0014] The NVMe instruction set is extended to add a conversion command, and the SSD executes the conversion locally after receiving the conversion command.

[0015] Further, the conversion command contains the following information: Src: the offset of the vertical stripe in the SSD; Dst: the logical address where the parity block will be stored; ChunkSize: the size of the block; Num: the number of data blocks in a stripe, that is, k; PID: the algorithm for calculating the parity block.

[0016] Further, after receiving the conversion command, the SSD executes the conversion locally, including: the SSD controller reads Num blocks of ChunkSize size from Src into its on-board memory, then calculates the parity block according to PID and writes the parity block into Dst, and finally the SSD clears the data in the vertical stripe by marking the flash page as invalid.

[0017] Further, the impact of SSD garbage collection on the performance of the full flash array is reduced through a garbage collection detection mechanism and a garbage collection avoidance mechanism.

[0018] Further, the garbage collection detection mechanism detects garbage collection in two ways: passive and active:

[0019] Passive mode: When the SSD firmware starts garbage collection, this information is encapsulated into the completion queue entry of the subsequent NVMe request to notify the host that the SSD has started garbage collection; after receiving the notification, the host avoids using this SSD to process I / O requests through the garbage collection avoidance mechanism, thereby alleviating the impact of the SSD garbage collection on the array performance;

[0020] Active mode: The host periodically sends NVMe commands to query whether the SSD has completed garbage collection. If so, the garbage collection avoidance mechanism for this SSD is lifted, and the host continues to use this SSD to complete I / O.

[0021] Further, the garbage collection avoidance mechanism includes:

[0022] In the copy stage, the host stops writing data to some stripe groups. If the vertical stripes that make up these stripe groups include blocks from the SSD that is undergoing garbage collection, new stripe groups are constructed. The vertical stripes in the new stripe groups only contain blocks from the SSDs that are not in the garbage collection process, and the data is written to the new stripe groups;

[0023] In the conversion stage, the array preferentially selects the vertical stripes in the SSDs that are not in the garbage collection process for conversion.

[0024] The beneficial effects of the present invention are as follows:

[0025] 1) Through the software-hardware co-design of the present invention, the conversion stage is offloaded to the SSD to complete, thus solving the problem of performance degradation caused by a large amount of I / O between the host and the SSD during this stage;

[0026] 2) Through the software-hardware co-design of the present invention, by garbage collection detection and garbage collection avoidance, the impact of SSD garbage collection on the array performance is alleviated. Description of the Drawings

[0027] Figure 1 is a schematic diagram of a two-stage write all-flash array.

[0028] Figure 2 is a schematic diagram of storage space abstraction.

[0029] Figure 3 is a schematic diagram of write path optimization.

[0030] Figure 4 is the experimental result of Key Technical Point 1.

[0031] Figure 5It is the experimental result of Key Technical Point 2. Detailed Implementation Manner

[0032] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below through specific embodiments and accompanying drawings.

[0033] 1. Storage Space Abstraction

[0034] Figure 2 It shows the abstraction of the storage space of the present invention. From the user's perspective, the present invention abstracts the all-flash array into a standard block device, exposing a continuous storage space, called the user address space, which supports random writes and in-place updates. The data written by the user will be sequentially recorded in an indirect address space, called the AFA address space. k consecutive data blocks (also called k slots) in the AFA address space form a stripe. Each stripe records the storage locations (in which SSD and the logical address on that SSD) of k data blocks and the corresponding m parity blocks in the SSD.

[0035] The present invention divides the storage space of the SSD into two virtual regions: the normal region and the transient region. The former is used to store long-term existing data (such as the converted data blocks), while the latter is used to store transient data (such as the copies of data blocks). The present invention organizes these two regions into horizontal stripes and vertical stripes respectively. In the normal region, k + m data blocks and parity blocks with the same SSD logical address from k + m SSDs are combined into a horizontal stripe (HH, for example, Figure 2 HH#1 in ), while in the transient region, k consecutive data blocks in the same SSD form a vertical stripe (VH, for example, VH#3-0 in SSD 2). In addition, the present invention combines a horizontal stripe and m vertical stripes from m different SSDs into a stripe group (HG), for example, HG#0 composed of HH#1 and VH#3-0. The present invention tracks the storage locations of data blocks and parity blocks through stripe groups.

[0036] 2. Write Path Optimization

[0037] 1) Copy Phase: Through the aforementioned storage space abstraction, data blocks can be placed in stripes so as to aggregate them in the array for conversion. Figure 3 It illustrates this process. In the copy phase, if a data block is written, it will be copied m + 1 times. One of them, called the primary copy, will be stored in the horizontal stripe, while the other m copies, called the secondary copies, will be written into m vertical stripes. Taking Figure 3For example, when the user writes to address 0x0, the data block is recorded in the first slot of the stripe corresponding to HG#1. Therefore, during the replication phase, the primary replica is written to the first slot of HH#1 (①), and the secondary replica is placed in the first slot of VH#3-0 (②). Repeating this process, the data blocks belonging to the same stripe will be aggregated into a vertical stripe (e.g., in VH#3-0).

[0038] 2) Conversion phase: Subsequently, when the host is idle or there is insufficient vertical stripe, the array starts the conversion phase and offloads the conversion task to the SSD (e.g., SSD 3). To make the SSD aware of this, the present invention extends the NVMe instruction set by adding a conversion command. The following information is included in this new command: Src: the offset of the vertical stripe in the SSD; Dst: the logical address where the parity block will be stored; ChunkSize: the size of the block; Num: the number of data blocks in a stripe (i.e., k); PID: the algorithm for calculating the parity block. For example, in the Reed-Solomon code, PID is the row number of the encoding vector in the Vandermonde matrix. After receiving the command, the SSD performs the conversion locally. Specifically, the SSD controller reads Num blocks of ChunkSize size from Src (③) into its on-board memory. Then it calculates the parity block according to PID (④) and writes the parity block to Dst (⑤). Finally, the SSD clears the data in the vertical stripe (e.g., VH#3-0) by marking the flash pages as invalid. Note that after step 4, the horizontal stripe (e.g., HH#1) is already in the stripe layout. Through these operations, the present invention successfully reduces the frequent I / O communication between the host and the SSD during the conversion phase, thus alleviating the problem of performance degradation in this phase.

[0039] 3. Reducing the impact of garbage collection

[0040] To mitigate the impact of SSD garbage collection on the array performance, the present invention adopts two designs: garbage collection detection and garbage collection avoidance.

[0041] 1) Garbage collection detection: This step is mainly to let the array know which SSD has started or ended garbage collection. Since the SSD firmware schedules each garbage collection event in the SSD, the present invention detects garbage collection in both passive and active ways through a software-hardware co-design. In the passive way, when the SSD firmware starts garbage collection, it encapsulates the garbage collection information into the completion queue entry (CQE) of the subsequent NVMe request to notify the host that the SSD has started garbage collection. After receiving the notification, the host avoids using this SSD to process I / O requests through the garbage collection avoidance mechanism, thereby alleviating the impact of the SSD's garbage collection on the array performance. In the active way, the host periodically sends NVMe commands to ask the SSD whether it has completed garbage collection. If so, the garbage collection avoidance mechanism for this SSD is lifted, and the host can continue to use this SSD to complete I / O.

[0042] 2) Garbage collection avoidance: In the copy phase, the host stops writing data to some stripe groups if the vertical stripes that make up these stripe groups include blocks from the SSD that is in garbage collection. Instead, new stripe groups are constructed, and the vertical stripes in these new stripe groups only contain blocks from the SSDs that are not in the garbage collection process. The array writes data to these new stripe groups. In the conversion phase, the array preferentially selects the vertical stripes in the SSDs that are not in the garbage collection process for conversion.

[0043] The key technical points of the present invention are as follows:

[0044] Key technical point one: Through software-hardware co-design, the conversion phase is offloaded to the SSD to complete, thus solving the problem of performance degradation caused by a large amount of I / O between the host and the SSD in this phase;

[0045] Key technical point two: Through software-hardware co-design, the impact of SSD garbage collection on the array performance is alleviated.

[0046] Advantages and beneficial effects of the present invention:

[0047] The inventor uses FEMU, an SSD emulator based on QEMU, to evaluate the software-hardware co-design in the present invention. The inventor sets the simulated SSD as a high-performance storage device with peak read and write bandwidths of 7500MB / s and 4890MB / s. These settings match those of commercial high-end SSD products. In addition, the inventor emulates an XOR engine in the SSD controller for calculating parity blocks. The inventor obtains the parameters of the XOR engine from a Xilinx xc7a200t FPGA with a clock frequency of 200MHz. It takes 20 microseconds for this XOR engine to calculate a 64KB parity block from 6 data blocks.

[0048] For the first key technical point, the experimental results show (as Figure 4 shown), compared with the existing solutions, in the experiments of 64KB random write and sequential write, and full stripe random write and sequential write, the write bandwidth of the present invention has increased by 36.1%, 39.6%, 34.0% and 37.9% respectively.

[0049] For the second key technical point, the experimental results show (as Figure 5 shown), compared with the existing solutions, the P9999 tail latency of the present invention in workloads such as src2 and DAP has been reduced by 8.9% and 20.5% respectively. In the microbenchmark, this improvement reaches 44.1%.

[0050] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and implement it accordingly. Those of ordinary skill in the art can understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification, and the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. An all-flash array based on software-hardware collaborative design, characterized in that: The all-flash array includes an AFA address space for sequentially recording data written by a user; k consecutive data blocks in the AFA address space form a stripe, and each stripe records the storage positions of k data blocks and corresponding m check blocks in the SSD; the storage space of the SSD in the all-flash array is divided into a normal area and a transient area, the normal area is used to store long-term data, and the transient area is used to store transient data; in the normal area, k+m data blocks and check blocks with the same SSD logical address from k+m SSDs are combined into a horizontal stripe; in the transient area, k consecutive data blocks in the same SSD form a vertical stripe; a horizontal stripe and m vertical stripes from m different SSDs are combined into a stripe group, and the storage positions of data blocks and check blocks are tracked through the stripe group.

2. The all-flash array based on software-hardware collaborative design according to claim 1, characterized in that: The all-flash array writes data blocks in two stages: The replication phase and the conversion phase.

3. The all-flash array based on software-hardware collaborative design according to claim 2, characterized in that: The replication phase includes: when writing a data block, the data block is replicated m+1 times, one of which is called the primary copy and is stored in the horizontal stripe, and the other m copies are called slave copies and are written into m vertical stripes.

4. The all-flash array based on software-hardware collaborative design according to claim 1, characterized in that: The conversion phase includes: When the host is idle or the vertical stripes are insufficient, the conversion phase begins and the conversion task is offloaded to the SSD; The NVMe instruction set is extended and a conversion command is added. After receiving the conversion command, the SSD performs the conversion locally.

5. The all-flash array based on software-hardware collaborative design according to claim 4, characterized in that: The conversion command includes the following information: Src: the offset of the vertical stripe in the SSD; Dst: the logical address where the check block will be stored; ChunkSize: the size of the block; Num: the number of data blocks in a stripe, that is, k; PID: Algorithm for calculating the check block.

6. The all-flash array based on software-hardware collaborative design according to claim 5, characterized in that: After receiving the conversion command, the SSD performs the conversion locally, including: the SSD controller reads Num blocks of size ChunkSize from Src into its onboard memory, then calculates the check block according to PID and writes the check block to Dst, and finally the SSD clears the data in the vertical stripe by marking the flash memory page as invalid.

7. The all-flash array based on software-hardware collaborative design according to claim 1, characterized in that: The impact of SSD garbage collection on the performance of all-flash arrays is mitigated through garbage collection detection mechanism and garbage collection avoidance mechanism.

8. The all-flash array based on software-hardware collaborative design according to claim 7, characterized in that: The garbage collection detection mechanism detects garbage collection in two ways: passive and active: Passive method: When the SSD firmware starts garbage collection, it encapsulates the garbage collection information into the completion queue entry of the subsequent NVMe request to notify the host that the SSD has started garbage collection. After receiving the notification, the host avoids using the SSD to process I / O requests through the garbage collection avoidance mechanism, thereby alleviating the impact of the SSD garbage collection on the array performance. Active mode: The host periodically sends NVMe commands to ask the SSD whether it has completed garbage collection. If so, the garbage collection avoidance mechanism for the SSD is released, and the host continues to use the SSD to complete I / O.

9. The all-flash array based on software-hardware collaborative design according to claim 7, characterized in that: The garbage collection avoidance mechanism includes: In the replication phase, the host stops writing data to certain stripe groups. If the vertical stripes constituting these stripe groups include blocks from SSDs that are undergoing garbage collection, a new stripe group is constructed. The vertical stripes in the new stripe group only contain blocks from SSDs that are not undergoing garbage collection, and the data is written to the new stripe group. During the conversion phase, the array preferentially selects vertical stripes in SSDs that are not in the garbage collection process for conversion.

Images