A data processing method and device based on large-scale storage system

Abstract

The present invention discloses a data processing method and device based on a large-scale storage system, and relates to the field of computer storage technology. The data processing method based on a large-scale storage system includes dividing the persistence layer of the large-scale storage system into a pre-write log module, an index memory cache module and an index persistence module, then writing data to the persistence layer through a write process, and then reading user data from the persistence layer through a read process. In a large-scale storage system based on a write-ahead log, the present invention optimizes the data reading and writing process of the persistence layer, improves the data double-writing problem caused by WAL, has less memory requirements, and does not increase unnecessary write delays due to the independent storage of data and index metadata; there is no need to significantly modify the WAL design, and for large I / O, only the index can be recorded, while for small I / O, user data can also be recorded in the index at the same time, reducing one data redirection, and supporting the optimization of I / O of different sizes.

Description

Technical Field

[0001] The present invention relates to the field of computer storage technology, and in particular to a data processing method and device based on a large-scale storage system. Background Art

[0002] Modern storage systems provide ACID transaction features. To ensure the atomicity and persistence in ACID, write-ahead logging (WAL) technology is commonly used. To modify user data in the storage system, the operation log is first appended to the write-ahead logging module, and then written to the persistence backend (such as disk blocks, data files, database engines, etc.).

[0003] WAL is the abbreviation of Write Ahead Logging, which records the modification operations on system data in the form of logs. The write-ahead log is a submodule of the persistence layer of the storage system. It is located at the forefront of the system write path and is used to record the operation log of system data modification. It can be used for crash recovery or transaction rollback. The log record is the smallest record unit of the write-ahead log file. Each record contains at least: CRC checksum, LSN log sequence number, Type operation type (modification or deletion), Size data payload size, Payload user data payload. To improve write performance and reliability, multiple records are continuously placed in data blocks and written in page alignment. Each data block can accommodate multiple log records, and the block size is fixed. The write-ahead log record and file structure are as follows: Figure 1 shown.

[0004] A WAL log record contains both modification operations and user data (Payload), and is arranged in the order of operations, so data rollback and recovery can be easily performed. If the log record is successful, it is considered that the data modification operation is successfully submitted to the system. Log records are written to the end of the WAL file in a sequential append manner, and the performance is higher than the random write method. User data is arranged in the order of writing in the WAL file, not in the order of the data primary key, and user data cannot be read randomly from the WAL file.

[0005] In a storage system based on write-ahead logs, WAL, as part of the system persistence layer, is located at the front end of the persistence layer and cooperates with the persistence backend to complete the atomic persistence of data. To persist data modification operations, they are first written to WAL and then finally written to the backend. To achieve crash recovery, WAL fully records the data operation log, and an operation log contains both data operations and user data. The persistence backend will store user data again, so there are actually two user data disk write operations, resulting in a data double write problem.

[0006] In large-scale data storage systems, WAL technology is needed to provide ACID transaction persistence and atomicity guarantees. In large-scale data storage systems, a typical storage node can manage tens of TB of user data, and the data read and write volume for each I / O is generally more than 4KB. The WAL data double-write problem reduces disk throughput and shortens disk life, causing great trouble to storage system design. Summary of the invention

[0007] Technical issues solved

[0008] In view of the deficiencies of the prior art, the present invention provides a data processing method and device based on a large-scale storage system, which solves the problem of double data writing caused by the introduction of WAL technology, while avoiding other problems of the prior art solution in the large-scale storage system scenario.

[0009] Technical Solution

[0010] To achieve the above objectives, the present invention is implemented through the following technical solutions: A data processing method based on a large-scale storage system, comprising the following specific method steps:

[0011] S1: Divide the persistence layer of the large-scale storage system into a write-ahead log module, an index memory cache module, and an index persistence module;

[0012] S2: Write data to the persistence layer through the write process;

[0013] S3: Read user data from the persistence layer through the read process.

[0014] Preferably, the write-ahead log module is located at the front end of the persistence layer, and the data writing operation is first recorded in the WAL. Each WAL record contains both the operation record and the user data. This behavior is consistent with the common WAL in the industry.

[0015] Preferably, after the data operation is recorded in WAL, data index information of "offset address of user data in WAL record in WAL file" needs to be established, and the index memory cache module is used to store the index information.

[0016] Preferably, the index persistence module is used to asynchronously write the data index information in the memory cache to the disk in B+tree or LSMtree format.

[0017] Preferably, the writing process includes the following specific steps:

[0018] S21: Append the data to the end of the current WAL file through the write-ahead log module, and obtain the WAL file name and the offset of the user data in the WAL file;

[0019] S22: if the additionally written data is difficult to retrieve, an index supporting random reading is established;

[0020] S23: The generated single index information is stored in the index memory cache module, and success can be immediately returned to the upper layer;

[0021] S24: The index information is asynchronously persisted to the disk in the B+tree or LSM tree format through the index persistence module;

[0022] S25: Redirect the data persisted to disk to cache, and then eliminate it from the memory cache or combine it with the LRU cache replacement algorithm to temporarily retain this part of data in the memory to speed up the next access.

[0023] Preferably, in said S24), when the system restarts normally or recovers from a crash, the data of the redirection index of the data that has not yet been persisted will be replayed, without traversing and loading all user data.

[0024] Preferably, in S23), only the index is recorded for the large I / O, while the user data is also recorded in the index for the small I / O, thereby reducing one data redirection.

[0025] Preferably, the reading process includes the following specific steps:

[0026] S31: first access the index memory cache module to query the specific offset of which WAL file the user data is located;

[0027] S32: If the cache does not hit, continue to query the index persistence module;

[0028] S33: If no user data information is found in the two index layers, all 0 data is returned;

[0029] S34: If a layer of indexes finds user data, the WAL file offset pointed to by the index is read to obtain the user data and return to the upper layer.

[0030] Preferably, a Bloom filter is set in S33) to filter user data that does not exist in the system to avoid cache penetration.

[0031] The present invention also provides a data processing device based on a large-scale storage system, comprising a system partitioning module, a data writing module and a data reading module, wherein:

[0032] System partitioning module, used to divide the persistence layer of large-scale storage systems into a write-ahead log module, an index memory cache module, and an index persistence module;

[0033] The data writing module is used to write data to the persistence layer through the writing process;

[0034] The data reading module is used to read user data from the persistence layer through the reading process.

[0035] Beneficial Effects

[0036] The present invention has the following beneficial effects:

[0037] (I) The data processing method and device based on the large-scale storage system regard the WAL file as a data file, and the user data is only recorded in the WAL, and is only written once, without double writing of data.

[0038] (ii) In the data processing method and device based on a large-scale storage system, metadata and user data are simultaneously stored on the disk in one I / O, thereby reducing latency.

[0039] (III) The data processing method and device based on a large-scale storage system combines the WAL file with the data redirection index. The data redirection index points to the user data in the WAL file, thereby solving the problem that the WAL file cannot be randomly read as a data file. The data redirection index is recorded in a combination of memory and disk, which reduces the memory space requirement. Moreover, after the system restarts normally or recovers from a crash, it is not necessary to completely traverse the data to rebuild the memory index.

[0040] (IV) The data processing method and device based on a large-scale storage system do not need to significantly modify the WAL design. For large I / Os, only the index can be recorded, while for small I / Os, user data can also be recorded in the index at the same time, thereby reducing one data redirection and supporting the optimization of I / Os of different sizes.

[0041] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 For existing write-ahead logging and file structures;

[0043] Figure 2 is a flow chart of the method of the present invention;

[0044] Figure 3 This is a schematic diagram of the persistence layer module of the present invention;

[0045] Figure 4 It is a schematic diagram of the data reading and writing process of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In the description of the present invention, it should be understood that the terms "opening", "upper", "lower", "thickness", "top", "middle", "length", "inside", "all around" and the like indicating orientation or positional relationship are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the components or elements referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0048] Example 1: Please refer to Figure 2-Figure 4 Based on the high load of large-scale data storage system in SSD scenario, the embodiment of the present invention provides a technical solution: a data processing method based on large-scale storage system, such as Figure 2 As shown, the specific method steps include the following:

[0049] S1: The persistence layer of the large-scale storage system is divided into a write-ahead log module, an index memory cache module, and an index persistence module, such as Figure 3 As shown, where:

[0050] The write-ahead log module is located at the front end of the persistence layer. Data write operations are first recorded in WAL. Each WAL record contains both operation records and user data. This behavior is consistent with the common WAL in the industry.

[0051] The index memory cache module, after the data operation is recorded in WAL, needs to establish data index information of "offset address of user data in WAL record in WAL file" for storing the index information;

[0052] The index persistence module is used to asynchronously write the data index information in the memory cache to the disk in the B+tree or LSM tree format;

[0053] S2: Write data to the persistence layer through the write process;

[0054] S3: Read user data from the persistence layer through the read process.

[0055] In this embodiment: the writing process includes the following specific steps: Figure 2 As shown:

[0056] S21: Append the data to the end of the current WAL file through the write-ahead log module, and obtain the WAL file name and the offset of the user data in the WAL file;

[0057] S22: If the appended data is difficult to retrieve, an index that supports random reading is established. The index is a Key-Value pair data structure, where the Key is the LBA and the Value is the offset address of the user data in the WAL file.

[0058] S23: The generated single index information is stored in the index memory cache module. At this time, success can be immediately returned to the upper layer. There is only one WAL write on the main I / O path for data writing, which is beneficial to the life of the SSD, and the low latency can provide higher performance.

[0059] S24: This index information is asynchronously persisted to disk in B+tree or LSM tree format through the index persistence module. When the system restarts normally or recovers from a crash, only the data of the redirection index that has not been persisted needs to be replayed, without traversing and loading all user data.

[0060] S25: Redirect the data persisted to disk to cache, and then eliminate it from the memory cache or combine it with the LRU cache replacement algorithm to temporarily retain this part of data in the memory to speed up the next access.

[0061] In this embodiment: in S23), only the index is recorded for the large I / O, while the user data is also recorded in the index for the small I / O, thereby reducing one data redirection.

[0062] In this embodiment, the reading process includes the following specific steps:

[0063] S31: first access the index memory cache module, and query the specific offset of the WAL file where the user data is located according to the LBA;

[0064] S32: If the cache does not hit, continue to query the index persistence module;

[0065] S33: If no user data information is found in the two-layer index, all 0 data is returned. A Bloom filter can be set to filter user data that does not exist in the system to avoid cache penetration;

[0066] S34: If a layer of indexes finds user data, the WAL file offset pointed to by the index is read to obtain the user data and return to the upper layer.

[0067] The present invention also provides a data processing device based on a large-scale storage system, comprising a system partitioning module, a data writing module and a data reading module, wherein:

[0068] System partitioning module, used to divide the persistence layer of large-scale storage systems into a write-ahead log module, an index memory cache module, and an index persistence module;

[0069] The data writing module is used to write data to the persistence layer through the writing process;

[0070] The data reading module is used to read user data from the persistence layer through the reading process.

[0071] Example 2: Please refer to Figure 2-Figure 4 Based on a large-scale Key-Value storage system scenario, an embodiment of the present invention provides a technical solution: a data processing method based on a large-scale storage system, such as Figure 2 As shown, the specific method steps include the following:

[0072] S1: The persistence layer of the large-scale storage system is divided into a write-ahead log module, an index memory cache module, and an index persistence module, such as Figure 3 As shown, where:

[0073] The write-ahead log module is located at the front end of the persistence layer. Data write operations are first recorded in WAL. Each WAL record contains both operation records and user data. This behavior is consistent with the common WAL in the industry.

[0074] The index memory cache module, after the data operation is recorded in WAL, needs to establish data index information of "offset address of user data in WAL record in WAL file" for storing the index information;

[0075] The index persistence module is used to asynchronously write the data index information in the memory cache to the disk in the B+tree or LSM tree format;

[0076] S2: Write data to the persistence layer through the write process;

[0077] S3: Read user data from the persistence layer through the read process.

[0078] In this embodiment: the Key-Value pair is written to the persistence layer, and the writing process includes the following specific steps: Figure 2 As shown:

[0079] S21: Append the Key-Value key value to the end of the current WAL file through the pre-write log module, and obtain the WAL file name and the offset of the user data in the WAL file;

[0080] S22: If the appended data is difficult to retrieve, an index that supports random reading is created, where the index primary key is the user Key and the index content is the offset address of the user Value in the WAL file;

[0081] S23: Store the generated single index information into the index memory cache module. At this time, success can be immediately returned to the upper layer. There is only one WAL write on the main I / O path for data writing, which is beneficial to the life of the SSD. The low latency can provide higher performance. In particular, for large key-value pairs exceeding 4K Value, the amount of data written can be greatly reduced, improving performance and disk life.

[0082] S24: This index information is asynchronously persisted to disk in B+tree or LSM tree format through the index persistence module. When the system restarts normally or recovers from a crash, only the data of the redirection index that has not been persisted needs to be replayed, without traversing and loading all user data.

[0083] S25: Redirect the data persisted to disk to cache, and then eliminate it from the memory cache or combine it with the LRU cache replacement algorithm to temporarily retain this part of data in the memory to speed up the next access.

[0084] In this embodiment: in S23), only the index is recorded for the large I / O, while the user data is also recorded in the index for the small I / O, thereby reducing one data redirection.

[0085] In this embodiment: to read the Key-Value from the persistence layer, the upper layer needs to specify the Key, and the reading process includes the following specific steps:

[0086] S31: first access the index memory cache module, and query the specific offset of the WAL file where the Key-Value is located according to the Key;

[0087] S32: If the cache does not hit, continue to query the index persistence module;

[0088] S33: If the user data information is not found in both indexes, all 0 data is returned. A Bloom filter can be set to filter the Key-Value that does not exist in the system to avoid cache penetration;

[0089] S34: If a layer of indexes finds a Key, the WAL file offset pointed to by the index is read, the Key-Value is obtained, and returned to the upper layer.

[0090] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0091] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method based on a large-scale storage system, characterized in that: The specific method steps include: S1: Divide the persistence layer of the large-scale storage system into a write-ahead log module, an index memory cache module, and an index persistence module; S2: Write data to the persistence layer through the write process; S3: Reads user data from the persistence layer through the read process; The writing process includes the following specific steps: S21: Append the data to the end of the current WAL file through the write-ahead log module, and obtain the WAL file name and the offset of the user data in the WAL file; S22: if the additionally written data is difficult to retrieve, an index supporting random reading is established; S23: The generated single index information is stored in the index memory cache module, and success can be immediately returned to the upper layer; S24: The index information is asynchronously persisted to the disk in the B+tree or LSM tree format through the index persistence module; S25: Redirect the data persisted to disk to cache, and then eliminate it from the memory cache or combine it with the LRU cache replacement algorithm to temporarily retain this part of data in the memory to speed up the next access.

2. The data processing method based on a large-scale storage system according to claim 1, characterized in that: The write-ahead log module is located at the front end of the persistence layer. The data writing operation is first recorded in the WAL. Each WAL record contains both the operation record and the user data. This behavior is consistent with the common WAL in the industry.

3. The data processing method based on a large-scale storage system according to claim 2, characterized in that: After the data operation is recorded in WAL, it is necessary to establish data index information of "the offset address of the user data in the WAL record in the WAL file", and the index memory cache module is used to store the index information.

4. The data processing method based on a large-scale storage system according to claim 1, characterized in that: The index persistence module is used to asynchronously write the data index information in the memory cache to the disk in the B+tree or LSM tree format.

5. The data processing method based on a large-scale storage system according to claim 1, characterized in that: In the S24), when the system restarts normally or recovers from a crash, the data of the redirection index of the data that has not yet been persisted will be replayed.

6. The data processing method based on a large-scale storage system according to claim 1, characterized in that: In the above S23), only the index is recorded for the large I / O, while the user data is also recorded in the index for the small I / O.

7. The data processing method based on a large-scale storage system according to claim 1, characterized in that: The reading process includes the following specific steps: S31: first access the index memory cache module to query the specific offset of the WAL file where the user data is located; S32: If the cache does not hit, continue to query the index persistence module; S33: If no user data information is found in the two index layers, all 0 data is returned; S34: If a layer of indexes finds user data, the WAL file offset pointed to by the index is read to obtain the user data and return to the upper layer.

8. The data processing method based on a large-scale storage system according to claim 7, characterized in that: In the S33), a Bloom filter is set to filter user data that does not exist in the system.

9. A data processing device based on a large-scale storage system, characterized in that: It includes a system partitioning module, a data writing module and a data reading module, wherein: System partitioning module, used to divide the persistence layer of large-scale storage systems into a write-ahead log module, an index memory cache module, and an index persistence module; The data writing module is used to write data to the persistence layer through the writing process; The data reading module is used to read user data from the persistence layer through the reading process; The writing process includes the following specific steps: S21: Append the data to the end of the current WAL file through the write-ahead log module, and obtain the WAL file name and the offset of the user data in the WAL file; S22: if the additionally written data is difficult to retrieve, an index supporting random reading is established; S23: The generated single index information is stored in the index memory cache module, and success can be immediately returned to the upper layer; S24: The index information is asynchronously persisted to the disk in the B+tree or LSM tree format through the index persistence module; S25: Redirect the data persisted to disk to cache, and then eliminate it from the memory cache or combine it with the LRU cache replacement algorithm to temporarily retain this part of data in the memory to speed up the next access.

Images