A data-reference-rate-based flash / scm hybrid array method, apparatus, and device

Abstract

The application provides a flash / SCM hybrid array method based on data reference rate, and relates to the field of computer storage. A RAID4 architecture for storing data in flash memory and storing check information in a new storage medium SCM is constructed, comprising: (1) initialization: the host module first initializes the flash memory and the SCM, and then initializes the metadata; (2) read / write request processing: after the first step of initialization, after receiving a read / write request, entering a read / write request processing module, which processes the read / write request initiated by the user; (3) data recovery: after the first step of initialization, after receiving a data recovery request, entering a data recovery module, which completes the data recovery work; (4) strip-based garbage collection; using the characteristics that the SCM has better durability and read / write performance than the flash medium, the overall performance of the storage system is improved. Using a strip-based garbage collection strategy, data blocks with high reference times are stored in the same strip, reducing write amplification.

Description

Technical Field

[0001] This invention relates to the field of storage, and in particular to a method, apparatus, and device for a flash / SCM hybrid array based on data reference rate. Background Technology

[0002] In recent years, solid-state drives (SSDs) have gradually become a powerful alternative to hard disk drives (HDDs) due to their advantages such as high read / write performance, low power consumption, good shock resistance, and increasingly lower prices. However, with the development of SSD technology, a single SSD is insufficient to meet the capacity, performance, and reliability requirements of enterprise applications. Therefore, it is necessary to apply RAID technology to flash storage systems. Due to the price of flash memory, parity redundancy protection mechanisms (RAID 4 / 5 / 6) are typically used to build all-flash arrays, and deduplication technology is used to improve storage efficiency and reduce costs.

[0003] However, all-flash arrays based on parity checking have some problems: parity updates cause write amplification, affecting the performance and reliability of the flash storage system. A single write request will update both the data block and the parity data block, resulting in logical write amplification. Moreover, the parity data block is a hot data block; updating any data block in the same band requires updating the parity block, significantly reducing the lifespan of the flash memory. With the application of TLC and QLC chips, the erase lifespan of flash memory is even shorter. Therefore, how to solve the write amplification problem caused by parity updates and improve performance and reliability has become an urgent problem to be solved.

[0004] Existing research uses a RAID 1 array of two disks as a cache pool for the solid-state drive (SSD) array, employing a log-based sequential write method and converting lowercase to uppercase to reduce the need for checksum updates. This method leverages the lossless nature of disk media to cache write data, but it significantly reduces system read performance, and the checksum update problem within the flash memory array remains unresolved. Summary of the Invention

[0005] The main objective of this invention is to overcome the aforementioned deficiencies in the prior art and propose a flash / SCM hybrid array method based on data reference rate. This method constructs a hybrid RAID4 architecture that uses flash memory to store data and SCM to store verification information, thereby improving read / write performance and durability without reducing system read / write performance or affecting the overall lifespan of the solid-state drive.

[0006] The present invention adopts the following technical solution:

[0007] A flash / SCM hybrid array method based on data reference rate, wherein the method uses flash memory to store data and SCM to store parity information to construct a RAID4 architecture, and a stripe-based garbage collection strategy; specifically including:

[0008] (1) Initialization: The main control module first initializes the flash memory and SCM, and then initializes the metadata;

[0009] (2) Read / write request processing: After initialization in the first step, upon receiving a read / write request, the process enters the read / write request processing module, which handles the read / write requests initiated by the user.

[0010] (3) Data recovery: After the initialization step, upon receiving a data recovery request, the system enters the data recovery module, which completes the data recovery process.

[0011] (4) Strip-based garbage collection: After the first initialization, when the remaining space of the solid disk is insufficient, the strip-based garbage collection module is entered, which completes the garbage collection work.

[0012] Specifically, the initialization step includes:

[0013] (1.1) Flash memory and SCM initialization: Perform erase operations on the flash memory and SCM media to ensure data consistency of the stripes; transition process (1.2);

[0014] (1.2) Metadata initialization: Initialize the Ref.Count value of all data stripes to 0, indicating that no data has been written to the stripe. During data recovery, stripes with Ref.Count = 0 will not be recovered.

[0015] Specifically, the read / write request processing steps are as follows:

[0016] (2.1) Determine if the incoming request type is a read request. If yes, proceed to procedure (2.2); otherwise, proceed to procedure (2.3).

[0017] (2.2) Directly perform address translation, and then split the request and send it to the corresponding flash memory device to read the data, thus completing the read request processing;

[0018] (2.3) First, perform address translation, then determine whether it is a lowercase request based on the array block size. If yes, proceed to process (2.4); otherwise, proceed to process (2.5).

[0019] (2.4) Read the old data and old verification information, and proceed to process (2.5);

[0020] (2.5) Calculate the new verification information, write the new data and the new verification information into the flash memory device, and complete the write request processing.

[0021] Specifically, the data recovery steps are as follows:

[0022] (3.1) Select the data stripes of the array in sequence and determine whether the end of the flash array stripe has been reached. If yes, proceed to process (3.5); otherwise, proceed to process (3.2).

[0023] (3.2) Scan the metadata information of the stripe and determine whether the reference count of the stripe is 0. If yes, proceed to process (3.3); otherwise, proceed to process (3.4).

[0024] (3.3) indicates that there is no data in this strip and no recovery is needed. Skip it and select the next strip; proceed to process (3.1);

[0025] (3.4) Reconstruct the data block of the faulty flash memory in the data stripe, and select the next stripe after completion; return to process (3.1);

[0026] (3.5) indicates that data recovery is complete and exit this step.

[0027] Specifically, the strip-based waste recycling step includes:

[0028] (4.1) Select the data stripe to be garbage collected, read the valid data pages (i.e., data pages with a data reference rate greater than 0) in sequence, and then determine whether the data reference rate of the data page is greater than 1. If yes, proceed to process (4.2); otherwise, proceed to process (4.3).

[0029] (4.2) Migrate the data page to region 1, and set the minimum data reference rate in the data page of this stripe to the data reference rate value of this stripe; proceed to process (4.4);

[0030] (4.3) Migrate the data page to region 2 and set the data reference rate of this stripe to 1; proceed to step (4.4);

[0031] (4.4) Determine if all data pages have been checked; if yes, proceed to process (4.5); otherwise, read the next valid page in the data stripe and proceed to process (4.2).

[0032] (4.5) Wipe the strip off, and the waste recycling step is complete.

[0033] Another aspect of the present invention provides a flash / SCM hybrid array device based on data reference rate, the device comprising: a main control module, a read / write request processing module, a data recovery module, and a stripe-based garbage collection module;

[0034] Main control module: Selects the initialization, read / write request processing, data recovery, and garbage collection processes based on the array's operating status;

[0035] Read / write request processing module: responsible for processing read / write requests from upper-layer applications, and sending the read / write requests to the corresponding solid-state drives for processing based on the specific location of the data;

[0036] Data recovery module: When a disk fails, it can recover the data from the failed disk.

[0037] The striped garbage collection module reclaims invalid data pages and migrates valid data pages based on data reference rate when the array's free space reaches a threshold.

[0038] In another aspect, the present invention provides a computer device including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, it performs the steps of the above-described flash memory / SCM hybrid array method based on data reference rate.

[0039] As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0040] This invention provides a flash / SCM hybrid array method based on data reference rate, relating to the field of computer storage. A RAID4 architecture is constructed using flash memory to store data and SCM (a novel storage medium) to store verification information, comprising: (1) Initialization: The main control module first initializes the flash memory and SCM, and then initializes the metadata; (2) Read / write request processing: After initialization in step one, upon receiving a read / write request, the module processes the user-initiated read / write requests; (3) Data recovery: After initialization in step one, upon receiving a data recovery request, the module completes the data recovery process; (4) Striped garbage collection: Utilizing the better durability and read / write performance of SCM compared to flash memory, the overall performance of the storage system is improved. A striped garbage collection strategy is used to store data blocks with high reference counts on the same stripe, reducing write amplification. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the device according to an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of the device flow of the present invention;

[0043] Figure 3 This is a schematic diagram of the initialization steps in an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram illustrating the read / write request processing steps in an embodiment of the present invention;

[0045] Figure 5This is a schematic diagram of the data recovery steps in an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the spatial division in the waste recycling module of this invention.

[0047] Figure 7 This is a schematic diagram of the strip-based waste recycling step in an embodiment of the present invention.

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0049] This embodiment describes a flash / SCM hybrid array method and apparatus based on data reference rate. The invention will be further described below with reference to the accompanying drawings.

[0050] Figure 1 This is a schematic diagram of the device according to an embodiment of the present invention. A RAID 4 architecture is constructed using a solid-state drive based on flash memory as the data disk and a novel non-volatile storage medium (SCM) as the parity disk. The flash memory stores data, and the SCM stores parity information. SCM: Storage Class Memory, refers to novel non-volatile storage media such as PCRAM, MRAM, and STT-RAM; RAID: Redundant Array of Independent Disks, a redundancy check array with parity codes.

[0051] Figure 2 This is a schematic diagram of the device of the present invention; after the main control module receives the read and write request issued by the application, it enters the flash memory controller, and the initialization module, array management module and garbage collection module process the request in sequence, and determine the layout and mapping of data in the array based on the stripes.

[0052] A flash / SCM hybrid array method based on data reference rate, wherein the method uses flash memory to store data and SCM to store parity information to construct a RAID4 architecture, and a stripe-based garbage collection strategy; specifically including:

[0053] (1) Initialization: The main control module first initializes the flash memory and SCM, and then initializes the metadata;

[0054] Specifically, the initialization step includes:

[0055] (1.1) Flash memory and SCM initialization: Perform erase operations on the flash memory and SCM media to ensure data consistency of the stripes; transition process (1.2);

[0056] 1. (1.2) Metadata initialization: Initialize the Ref.Count value of all data stripes to 0, indicating that no data has been written to the stripe. During data recovery, stripes with Ref.Count = 0 will not be recovered. Ref.Count: Reference.Count, the data reference rate, which is the number of times a data block is referenced in the deduplication system.

[0057] (2) Read / write request processing: After initialization in the first step, upon receiving a read / write request, the process enters the read / write request processing module, which handles the read / write requests initiated by the user.

[0058] like Figure 3 This is a schematic diagram of the initialization steps in an embodiment of the present invention. The initialization module first performs flash memory and SCM initialization, and performs erase operations (such as writing 0) on both flash memory and SCM media to ensure data consistency across all stripes (this is because traditional RAID4 / 5 initialization requires reading data and then reconstructing the parity and writing it to the parity disk, which is more complex). Then, it initializes the metadata, that is, initializes the reference rate value of all data stripes to 0, indicating that no data has been written to the stripe, ensuring data consistency across all stripes. During data recovery, stripes with a reference rate of 0 are not recovered.

[0059] Specifically, the read / write request processing steps are as follows:

[0060] (2.1) Determine if the incoming request type is a read request. If yes, proceed to procedure (2.2); otherwise, proceed to procedure (2.3).

[0061] (2.2) Directly perform address translation, and then split the request and send it to the corresponding flash memory device to read the data, thus completing the read request processing;

[0062] (2.3) First, perform address translation, then determine whether it is a lowercase request based on the array block size. If yes, proceed to process (2.4); otherwise, proceed to process (2.5).

[0063] (2.4) Read the old data and old verification information, and proceed to process (2.5);

[0064] (2.5) Calculate the new verification information, write the new data and the new verification information into the flash memory device, and complete the write request processing.

[0065] Figure 4This is a schematic diagram of the read / write request processing steps in an embodiment of the present invention. The process involves determining whether the incoming request is a read request. If it is, address translation is performed directly, and the request is segmented and sent to the corresponding flash memory device to read the data, thus completing the read request processing. Otherwise, after address translation, the process determines whether it is a lowercase request based on the array block size. If it is a lowercase request, the old data and old checksum are read first, then new checksum is calculated, and the new data and new checksum are written to the flash memory device. Otherwise, new checksum is calculated, and the new data and new checksum are written to the flash memory device, thus completing the write request processing.

[0066] (3) Data recovery: After the initialization step, upon receiving a data recovery request, the system enters the data recovery module, which completes the data recovery process.

[0067] Specifically, the data recovery steps are as follows:

[0068] (3.1) Select the data stripes of the array in sequence and determine whether the end of the flash array stripe has been reached. If yes, proceed to process (3.5); otherwise, proceed to process (3.2).

[0069] (3.2) Scan the metadata information of the stripe and determine whether the reference count of the stripe is 0. If yes, proceed to process (3.3); otherwise, proceed to process (3.4).

[0070] (3.3) indicates that there is no data in this strip and no recovery is needed. Skip it and select the next strip; proceed to process (3.1);

[0071] (3.4) Reconstruct the data block of the faulty flash memory in the data stripe, and select the next stripe after completion; return to process (3.1);

[0072] (3.5) indicates that data recovery is complete and exit this step.

[0073] like Figure 5 This is a schematic diagram of the data recovery steps in this embodiment of the invention; the data stripes of the array are selected sequentially, and it is determined whether the end of the flash array strip has been reached. If so, it indicates that the data recovery is complete and the step is exited; if not, the metadata information of the stripe is scanned to determine whether the reference count of the stripe is 0. If so, it indicates that the stripe has no data and does not need to be recovered, so it is skipped and the next stripe is selected; otherwise, the data block of the faulty flash memory in the data stripe is reconstructed, and the next stripe is selected after completion.

[0074] (4) Strip-based garbage collection: After the first initialization, when the remaining space of the solid disk is insufficient, the strip-based garbage collection module is entered, which completes the garbage collection work.

[0075] Specifically, the strip-based waste recycling step includes:

[0076] (4.1) Select the data stripe to be garbage collected, read the valid data pages (i.e., data pages with a data reference rate greater than 0) in sequence, and then determine whether the data reference rate of the data page is greater than 1. If yes, proceed to process (4.2); otherwise, proceed to process (4.3).

[0077] (4.2) Migrate the data page to region 1, and set the minimum data reference rate in the data page of this stripe to the data reference rate value of this stripe; proceed to process (4.4);

[0078] (4.3) Migrate the data page to region 2 and set the data reference rate of this stripe to 1; proceed to step (4.4);

[0079] (4.4) Determine if all data pages have been checked; if yes, proceed to process (4.5); otherwise, read the next valid page in the data stripe and proceed to process (4.2).

[0080] (4.5) Wipe the strip off, and the waste recycling step is complete.

[0081] Figure 6 This is a schematic diagram of spatial region division based on the strip garbage collection step in an embodiment of the present invention; similar data pages are placed in the same strip according to the reference value of the data page. The green dashed area represents the area where the strip reference rate is greater than 1, the red dashed area represents the area where the strip reference rate is equal to 1, and the black dashed area represents the area where the strip reference rate is equal to 0.

[0082] Figure 7 This is a schematic diagram of the stripe-based garbage collection step in an embodiment of the present invention. The data stripe to be garbage collected is selected, and valid data pages (i.e., data pages with a data reference rate greater than 0) are read sequentially. Then, it is determined whether the data reference rate of the data page is greater than 1. If so, the data page is migrated to region 1, and the minimum data reference rate in the data page of that stripe is set as the data reference rate value of that stripe. Otherwise, the data page is migrated to region 2, and the data reference rate value of that stripe is set to 1. Then, it is determined whether all data pages have been checked. If so, the stripe is erased, and the garbage collection step ends; otherwise, the next valid page in the data stripe is read, and this process continues until all pages have been checked.

[0083] Another aspect of the present invention provides a flash / SCM hybrid array device based on data reference rate, the device comprising: a main control module, a read / write request processing module, a data recovery module, and a stripe-based garbage collection module;

[0084] Main control module: Selects the initialization, read / write request processing, data recovery, and garbage collection processes based on the array's operating status;

[0085] Read / write request processing module: responsible for processing read / write requests from upper-layer applications, and sending the read / write requests to the corresponding solid-state drives for processing based on the specific location of the data;

[0086] Data recovery module: When a disk fails, it can recover the data from the failed disk.

[0087] The striped garbage collection module reclaims invalid data pages and migrates valid data pages based on data reference rate when the array's free space reaches a threshold.

[0088] This invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a method for a flash memory / SCM hybrid array based on data reference rate provided in this invention.

[0089] Since the electronic device described in this embodiment is the device used to implement the embodiments of the present invention, those skilled in the art can understand the specific implementation methods and various variations of the electronic device in this embodiment based on the methods described in the embodiments of the present invention. Therefore, how the electronic device implements the methods in the embodiments of the present invention will not be described in detail here. Any device used by those skilled in the art to implement the methods in the embodiments of the present invention is within the scope of protection of the present invention.

[0090] This invention provides a flash / SCM hybrid array method based on data reference rate, relating to the field of computer storage. A RAID4 architecture is constructed using flash memory to store data and SCM (a novel storage medium) to store verification information, comprising: (1) Initialization: The main control module first initializes the flash memory and SCM, and then initializes the metadata; (2) Read / write request processing: After initialization in step one, upon receiving a read / write request, the module processes the user-initiated read / write requests; (3) Data recovery: After initialization in step one, upon receiving a data recovery request, the module completes the data recovery process; (4) Striped garbage collection: Utilizing the better durability and read / write performance of SCM compared to flash memory, the overall performance of the storage system is improved. A striped garbage collection strategy is used to store data blocks with high reference counts on the same stripe, reducing write amplification.

[0091] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A method for a hybrid flash / SCM array based on data reference rate, characterized in that, The method employs Flash memory to store data and SCM to store verification information to construct a RAID4 architecture, along with a stripe-based garbage collection strategy; specifically, it includes: (1) Initialization: The main control module first initializes the flash memory and SCM, and then initializes the metadata; (2) Read / write request processing: After initialization in step (1), upon receiving a read / write request, the read / write request processing module is entered, which processes the read / write requests initiated by the user. (3) Data recovery: After initialization in step (1), upon receiving a data recovery request, the system enters the data recovery module, which completes the data recovery process. (4) Strip-based garbage collection: After initialization in step (1), when the remaining space of the solid disk is insufficient, the strip-based garbage collection module is entered, which completes the garbage collection work; The strip-based waste recycling specifically refers to: (4.1) Select the data stripe to be garbage collected, and read the valid data pages in it in sequence. The valid data pages are data pages with a data reference rate greater than 0. Then determine whether the data reference rate of the data page is greater than 1. If yes, proceed to process (4.2); otherwise, proceed to process (4.3). (4.2) Migrate the data page to region 1 and set the minimum data reference rate in the data page of this stripe to the data reference rate value of this stripe; proceed to process (4.4). (4.3) Move the data page to region 2 and set the data reference rate of the stripe to 1; proceed to process (4.4). (4.4) Determine whether all data pages have been checked; if yes, proceed to process (4.5); otherwise, read the next valid page in the data stripe and proceed to process (4.2). (4.5) Wipe off the strip, and the waste recycling step is complete.

2. The flash memory / SCM hybrid array method based on data reference rate according to claim 1, characterized in that, The initialization specifically includes: (1.1) Flash memory and SCM initialization: Perform erase operations on the flash memory and SCM media to ensure data consistency of the stripes; Transformation process (1.2); (1.2) Metadata initialization: Initialize the Ref. Count value of all data stripes to 0, indicating that no data has been written to the stripe. During data recovery, stripes with Ref. Count = 0 will not be recovered.

3. The flash / SCM hybrid array method based on data reference rate according to claim 1, characterized in that, The read / write request processing is specifically as follows: (2.1) Determine whether the incoming request type is a read request. If yes, proceed to process (2.2); otherwise, proceed to process (2.3). (2.2) Directly perform address translation, and then split the request and send it to the corresponding flash memory device to read the data, thus completing the read request processing; (2.3) First, perform address translation, then determine whether it is a lowercase request based on the array block size. If yes, proceed to process (2.4); otherwise, proceed to process (2.5). (2.4) Read the old data and old verification information, and proceed to process (2.5); (2.5) Calculate the new verification information, write the new data and the new verification information into the flash memory device, and complete the write request processing.

4. The flash / SCM hybrid array method based on data reference rate according to claim 1, characterized in that, The data recovery specifically involves: (3.1) Select the data stripes of the array in sequence and determine whether the end of the flash array stripe has been reached. If yes, proceed to process (3.5); otherwise, proceed to process (3.2). (3.2) Scan the metadata information of the stripe and determine whether the reference count of the stripe is 0. If yes, proceed to process (3.3); otherwise, proceed to process (3.4). (3.3) indicates that there is no data in this strip and no recovery is needed. Skip it and select the next strip; proceed to process (3.1). (3.4) Reconstruct the data block of the faulty flash memory in the data stripe, and select the next stripe after completion; proceed to process (3.1). (3.5) indicates that data recovery is complete and exit this step.

5. A flash memory / SCM hybrid array device based on data reference rate, characterized in that, Based on the data reference rate-based flash / SCM hybrid array method according to claim 1, the device includes: a main control module, a read / write request processing module, a data recovery module, and a stripe-based garbage collection module; Main control module: Selects the initialization, read / write request processing, data recovery, and garbage collection processes based on the array's operating status; Read / write request processing module: responsible for processing read / write requests from upper-layer applications, and sending the read / write requests to the corresponding solid-state drives for processing based on the specific location of the data; Data recovery module: Recovers data from a failed disk when a disk fails. The striped garbage collection module reclaims invalid data pages and migrates valid data pages based on data reference rate when the array's free space reaches a threshold.

6. A computer device, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it performs the steps of the method as described in any one of claims 1-4.

Images