Key-Value local storage method and system based on solid state disk (SSD)

Abstract

Disclosed are an SSD-based Key-Value type local storage method and system. The method comprises: step 1, performing a memory read-write separation operation on data using an index structure of a memory snapshot B+ tree; step 2, the indexed data using an FIFO queue to manage a cache for the B+ tree; and step 3, performing a read-write operation on the data, and achieving the mapping management of logical page numbers and physical locations in the log-type additionally written data through an empty file mechanism.

Description

technical field [0001] The invention relates to a local data storage management system, in particular to a Key-Value (key-value) type local storage method and system based on SSD (Solid State Disk). Background technique [0002] The organization and management of data is mainly divided into three steps. The first is online access to data, which mainly refers to obtaining data and providing reading services, that is, for traditional OLTP loads. The second is data organization, which traditionally refers to the The data in the OLTP database is converted into a data format suitable for the data warehouse, which is a process called ETL. The third is data analysis, which refers to long-term, complex data mining and other work to discover the connection and potential value in the data, that is, OLAP-type tasks. In this paper, we focus on the online access part of the data. [0003] In the traditional solution, the relational database represented by MySQL satisfies the task of on...

AI Technical Summary

Problems solved by technology

But for new loads in the era of big data, relational databases show their inherent limitations:

The first reason is that relatively speaking, there is no strict requirement for the characteristics of ACID in the new business requirements typical of Internet applications. For example, there are temporary inconsistencies in blog posts, related comments, album pictures, and even online store inventory. status is acceptable to the user

Second, the strict ACID feature restrictions make it difficult to improve the overall performance and scalability of the database, which is mainly caused by complex locking mechanisms, log mechanisms, etc.

[0014] Among the existing similar systems, systems such as Flashstore and FAWN use the Hash-type data indexing mechanism. There are two main problems in this indexing method. It is difficult to achieve the effect of both

Second, the Hash data index is difficult to implement range search operations

[0015] For many systems using the traditional B+ tree index mechanism represented by Berkeley DB, the main problem when using it on SSDs is that data insertion will cause a large number of in-place update and write operations, which is an IO mode that is not conducive to SSD performance. , on the other hand, for concurrency support, the B+ tree index needs to introduce a complex locking mechanism, which is not conducive to the overall performance of the system

[0016] The LSM-tree index structure that appeared later is applied in systems such as LevelDB. Its advantage is that the writing mode is large-grained continuous writing, which is very conducive to the performance of SSDs. However, LSM-tree is a mechanism that tends to write optimization. , the performance of the read operation is low due to the large number of hard disk reads introduced

[0017] To sum up, the existing Key-Value system cannot meet the current application requirements, mainly reflected in the following two points:

[0018] First, the concurrency control technology based on the lock mechanism is difficult to meet the requirements of high concurrent read and write load;

[0019] Second, the writing mode of the existing system does not adapt to the characteristics of SSD

Images