A method and system for increasing memory based on rocksdb storage engine

Abstract

The present invention discloses a method and system for increasing memory based on a rocksdb storage engine, and belongs to the field of cloud computing storage; the S1 utilizes a memory pool and a memory pool management mechanism to maintain data in a memory; S2 utilizes an adaptive radix tree index to replace the original index structure with data in a memory storage area; S3 utilizes a modified pre-write management system to manage multiple log files to realize their recovery function; S4 performs a function of writing data in the memory storage area to a disk, and manages the generated SST file; the purpose of the present invention is to provide a method and system for increasing memory based on a rocksdb storage engine, and while increasing the amount of storage data in the memory of the rocksdb storage engine, each subsystem in the rocksdb system is adapted and adjusted to achieve the goal of increasing memory and improving storage engine performance.

Description

Technical Field

[0001] The present invention discloses a method and system for increasing memory based on a rocksdb storage engine, and relates to the technical field of cloud computing storage. Background Art

[0002] In this era of rapid development, Internet technology has been further developed. Information technology has a significant role in promoting social economy, and the cornerstone of information technology is database technology, which is the carrier of information storage. Traditional disk databases have always played an important role in database technology and have achieved good results in traditional OLTP scenarios. However, people have higher and higher requirements for application response speed. In many high-real-time scenarios, traditional disk databases do not perform well. With the development of memory technology, the capacity of memory is getting larger and larger, and the price is getting lower and lower. Its superior access speed has made memory databases begin to be widely used in various scenarios. Memory capacity continues to increase, prices continue to fall, and operating systems can already support larger address spaces (computers have entered the 64-bit era). At the same time, the requirements for real-time response capabilities of database systems are increasing. Making full use of memory technology to improve database performance has become a hot topic.

[0003] However, there are still many problems that need to be solved in memory databases, including data security, rapid system recovery, and crazy memory usage. At the same time, it is necessary to adapt to different systems and environments. In order to increase memory for faster storage and reading speeds, it is necessary to consider the defects of volatile memory itself, such as volatility, system security, and cost trade-offs.

[0004] Therefore, a method and system for increasing memory based on the rocksdb storage engine is invented to solve the above problems. Summary of the invention

[0005] In view of the problems of the prior art, the present invention provides a method and system for increasing memory based on the rocksdb storage engine. The technical solution adopted is: a method for increasing memory based on the rocksdb storage engine, wherein the specific steps of the method are as follows:

[0006] S1 uses memory pool and memory pool management mechanism to maintain data in memory;

[0007] S2 uses adaptive radix tree index to replace the original index structure with data in the memory storage area;

[0008] S3 uses a modified pre-write management system to manage multiple log files to achieve its recovery function;

[0009] S4 stores the data in the memory storage area on disk and manages the generated SST files.

[0010] The specific steps of S1 using the memory pool and memory pool management mechanism to maintain the data in the memory are as follows:

[0011] S101 uses the memory pool to adapt according to the current amount of data in the database;

[0012] S102 stores and manages the data in the memory pool through the memory storage area.

[0013] The specific steps of S2 using the adaptive radix tree index to replace the original index structure with data in the memory storage area are as follows:

[0014] S201 uses the data structure of the adaptive radix tree as an index of the data in the memory storage area;

[0015] S202 uses the address and length of each piece of data in the memory storage area as a leaf node of the adaptive radix tree index;

[0016] S203 finds the corresponding leaf node through the index, and finds the data through the leaf node;

[0017] S204 uses a common database structure to adapt the adaptive radix tree.

[0018] The specific steps of S3 using the modified pre-write management system to manage multiple log files and realize its recovery function are as follows:

[0019] S301 manages multiple pre-written log files through component transformation;

[0020] S302 restores all files of the pre-write management system to the memory storage area.

[0021] The specific steps of the S4 for writing the data in the memory storage area to the disk and managing the generated SST file are as follows:

[0022] The data in the memory storage area of ​​S401 is sequentially written to the SST and the SST is split, and the SST is added to level 1;

[0023] S402 switches the storage engine that increases memory to the rocksdb storage engine.

[0024] A system for increasing memory based on the rocksdb storage engine, the system specifically includes a memory storage module, an ART index module, a pre-write management module and an SST data disk module:

[0025] Memory storage module: uses memory pool and memory pool management mechanism to maintain data in memory;

[0026] ART index module: uses adaptive radix tree index to replace the original index structure with data in the memory storage area;

[0027] Pre-write management module: Use the modified pre-write management system to manage multiple log files and realize their recovery function;

[0028] SST data disk module: It has the function of storing data in the memory storage area on disk and managing the generated SST files.

[0029] The memory storage module specifically includes an adaptive module and a storage management module:

[0030] Adaptive module: Adapts based on the current amount of data in the database through the memory pool;

[0031] Storage management module: stores and manages the data in the memory pool through the memory storage area.

[0032] The ART index module specifically includes an index establishment module, a node establishment module, a data index module and an index adaptation module:

[0033] Index building module: using the data structure of the adaptive radix tree as the index of the data in the memory storage area;

[0034] Node creation module: takes the address and length of each piece of data in the memory storage area as the leaf node of the adaptive radix tree index;

[0035] Data index module: find the corresponding leaf node through the index, and find the data through the leaf node;

[0036] Index adaptation module: Use the common structure of the database to adapt the adaptive radix tree.

[0037] The pre-write management module specifically includes a file management module and a file recovery module:

[0038] File management module: manage multiple pre-written log files through component transformation;

[0039] File recovery module: restores all files of the pre-written management system to the memory storage area.

[0040] The SST data storage module specifically includes a data segmentation module and an engine switching module:

[0041] Data segmentation module: The data in the memory storage area is sequentially stored in the SST and the SST is segmented, and the SST is added to level 1;

[0042] Engine switching module: Switch the storage engine with increased memory to the rocksdb storage engine.

[0043] The beneficial effects of the present invention are as follows:

[0044] The memory in the storage engine is increased, which speeds up the reading and writing of the latest data. A large amount of data is cached in the memory. During the reading process, the number of file reads is reduced, which can effectively improve the overall reading performance of the storage engine. A large amount of data cached in the memory can be written to the SST at one time, and the multiple SST files generated can ensure order. Compared with the previous situation of generating disordered SST files, the number and complexity of compaction can be reduced, thereby indirectly improving the writing speed of the memory engine.

[0045] Small memory waste: Using the ART index structure, a reasonable memory space can be adaptively allocated, and there will be no space waste caused by pre-allocation of memory, thus reducing the waste of memory space;

[0046] Make full use of spatial locality and temporal locality; through the buffer pool and buffer pool management mechanism of the memory storage area, the data is stored in the corresponding buffer pool in an orderly and compact manner, which can effectively utilize spatial locality and temporal locality, thereby improving the loading and storage performance of the storage engine;

[0047] Data is secure and recoverable. By transforming the logs of the rocksdb storage engine, a multi-file pre-write management system is implemented to ensure the security of memory data.

[0048] The engine is highly switchable. Since the SST file format remains the same as before, the SST file can be easily loaded into the rocksdb storage engine. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a structural schematic diagram of the system of the present invention; Figure 3 2 is a schematic diagram of speed comparison of an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention. Example

[0052] A method for increasing memory based on the rocksdb storage engine, wherein the specific steps of the method are as follows:

[0053] S1 uses memory pool and memory pool management mechanism to maintain data in memory;

[0054] S2 uses adaptive radix tree index to replace the original index structure with data in the memory storage area;

[0055] S3 uses a modified pre-write management system to manage multiple log files to achieve its recovery function;

[0056] S4 stores the data in the memory storage area on disk and manages the generated SST files;

[0057] First, according to S1, the memory storage area includes multiple memory pools and a memory pool management mechanism to maintain the data in the memory; then according to S2, the data structure of the adaptive radix tree is used as the index of the memory storage area. The leaf node of the adaptive radix tree is the address and length of a piece of data in the memory storage area; in order to ensure the security of a large amount of memory, according to S3, the modified pre-write management system is used to manage multiple log files, and the recovery function of multiple log files is realized; finally, according to S4, the data in the memory storage area is written to the disk, and the generated SST file is managed;

[0058] Furthermore, the specific steps of S1 maintaining the data in the memory using the memory pool and the memory pool management mechanism are as follows:

[0059] S101 uses the memory pool to adapt according to the current amount of data in the database;

[0060] S102 stores and manages the data in the memory pool through the memory storage area;

[0061] C++ provides new / delete to manage memory application and release, but for small objects, using new / delete directly is expensive, requiring extra space and time, and is not cost-effective. In addition, we must also avoid memory fragmentation caused by multiple applications and releases. Once the fragmentation reaches a certain level, even if the total amount of remaining memory is sufficient, the lack of sufficient continuous free space will lead to the illusion that the memory is insufficient. Therefore, we have implemented our own simple and efficient memory pool and managed the memory pool. The memory pool and the management of the memory pool are the core content of the memory storage area module.

[0062] The memory pool of the memory storage area is a character array of a customizable size, with a default size of 32M. The size of the memory pool can also be adapted according to the current amount of data in the database. The data is stored in a continuous memory space, and the indexing function of the ART tree index module obtains the starting position and data length of each piece of data, thereby reasonably arranging the storage and reading of the memory pool.

[0063] The management of the memory storage area mainly includes data migration management of the memory pool, creation and deletion management of the memory pool, and data storage and reading management of the memory pool; in order to improve the spatial locality and temporal locality of the memory storage, the S102 memory storage area stores and manages the data of the memory pool; in the memory pool of the memory storage area, the data is partially ordered, and the partially ordered adjacent data is stored in the same memory pool as much as possible; when the data in the memory pool reaches the threshold, a new memory pool is created and part of the data in the original memory pool is migrated; data migration management enables the memory storage area to always maintain efficient spatial locality and temporal locality;

[0064] Furthermore, the specific steps of S2 using the adaptive radix tree index to replace the original index structure with data in the memory storage area are as follows:

[0065] S201 uses the data structure of the adaptive radix tree as an index of the data in the memory storage area;

[0066] S202 uses the address and length of each piece of data in the memory storage area as a leaf node of the adaptive radix tree index;

[0067] S203 finds the corresponding leaf node through the index, and finds the data through the leaf node;

[0068] S204 uses a common database structure to adapt the adaptive radix tree;

[0069] Adaptive Radix Tree (ART) is used as an index to increase memory. Each node of the radix tree can store key slices of any length. For example, each node of the radix tree in the Linux Kernel stores a 6-bit key slice. However, database indexes are frequently modified in many scenarios, and fixed-length key slices for each node will waste time (slices are too long) and space (slices are too short). Therefore, we choose an adaptive radix tree, that is, a radix tree in which nodes are divided into several types according to their lengths and adjust themselves as data changes. Spin-lock modification of ART can achieve efficient concurrent management, thereby increasing the speed of concurrent insertion and query. ART is an upgraded version of the radix tree, and it proposes a solution to the inefficiency of the radix tree in cache hits. At the same time, delayed expansion and path compression are used to reduce the tree height, which can reduce the tree height in the presence of abnormal data. Figure 3 As shown, it still has a great advantage in speed. In the case of dense, the speed is the same as that of the hash table, and in the case of sparse, the speed is slightly faster than FAST (FAST is an algorithm optimized for specific hardware). In terms of memory usage, the structure of ART has 4 different nodes. ART can adaptively select nodes of different sizes to reduce the memory usage. We encapsulate and adapt the ART index module and use the ART index module as the underlying index of the memory storage area. We can sort the pointers to the data in the memory storage area through the ART index. In addition to providing seek and get functions, ART also has the function of range traversal.

[0070] Furthermore, the specific steps of S3 managing multiple log files using the modified pre-write management system to achieve its recovery function are as follows:

[0071] S301 manages multiple pre-written log files through component transformation;

[0072] S302 restores all files of the pre-write management system to the memory storage area;

[0073] RocksDB's write-ahead logging (WAL) management system is to ensure the security of rocksdb's memtable data. Since we increase the memory based on rockdb, we no longer use memtable, but use the memory storage area. Therefore, we need to transform the rocksdb write-ahead management system. The write-ahead management system will use WAL files to ensure the security of the data in the memory. Due to the increase in memory, there will be multiple WAL files in the write-ahead management system at the same time. We transform the maintenance queues of logs_ and alive_log_files_ in db_impl so that the write-ahead management system can maintain multiple WAL files. When writing WAL files, we cut the files according to the size of the WAL files to avoid generating large files. When restoring WAL, we change from restoring only one WAL file to traversing all WAL files in the current file system and restoring all WAL files to the memory storage area.

[0074] Furthermore, the S4 performs the function of writing the data in the memory storage area to the disk and manages the generated SST file in the following specific steps:

[0075] The data in the memory storage area of ​​S401 is sequentially written to the SST and the SST is split, and the SST is added to level 1;

[0076] S402 switches the storage engine that increases memory to the rocksdb storage engine;

[0077] Since we no longer use memtable and immutable memtable after increasing the memory, the original flush mechanism of rocksdb cannot be applied to the current increased memory. Taking advantage of the orderliness of the data in the memory storage area, we traverse the data range of the memory storage area through the ART index and write the data in the memory storage area to the SST file. We split the SST file by using a customer-defined threshold or according to the start key and end key of the level 2 SST file to avoid the SST file being too large and avoid generating large SST files during the compaction process. The files of the SST data disk module are ordered, which is different from the level 0 SST files generated by the original rocksdb flush mechanism. Therefore, we directly skip the level 0 SST files and directly generate the level 1 SST files. Example

[0078] A system for increasing memory based on the rocksdb storage engine, the system specifically includes a memory storage module, an ART index module, a pre-write management module and an SST data disk module:

[0079] Memory storage module: uses memory pool and memory pool management mechanism to maintain data in memory;

[0080] ART index module: uses adaptive radix tree index to replace the original index structure with data in the memory storage area;

[0081] Pre-write management module: Use the modified pre-write management system to manage multiple log files and realize their recovery function;

[0082] SST data storage module: It stores the data in the memory storage area on the disk and manages the generated SST files;

[0083] Furthermore, the memory storage module specifically includes an adaptive module and a storage management module:

[0084] Adaptive module: Adapts based on the current amount of data in the database through the memory pool;

[0085] Storage management module: stores and manages the data in the memory pool through the memory storage area;

[0086] Furthermore, the ART index module specifically includes an index establishment module, a node establishment module, a data index module and an index adaptation module:

[0087] Index building module: using the data structure of the adaptive radix tree as the index of the data in the memory storage area;

[0088] Node creation module: takes the address and length of each piece of data in the memory storage area as the leaf node of the adaptive radix tree index;

[0089] Data index module: find the corresponding leaf node through the index, and find the data through the leaf node;

[0090] Index adaptation module: Use the common structure of the database to adapt the adaptive radix tree.

[0091] Furthermore, the pre-writing management module specifically includes a file management module and a file recovery module:

[0092] File management module: manage multiple pre-written log files through component transformation;

[0093] File recovery module: restore all files in the pre-written management system to the memory storage area;

[0094] Furthermore, the SST data storage module specifically includes a data segmentation module and an engine switching module:

[0095] Data segmentation module: The data in the memory storage area is sequentially stored in the SST and the SST is segmented, and the SST is added to level 1;

[0096] Engine switching module: Switch the storage engine with increased memory to the rocksdb storage engine.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for increasing memory based on the rocksdb storage engine, characterized by: The specific steps of the method are as follows: S1 uses memory pool and memory pool management mechanism to maintain data in memory; S2 uses adaptive radix tree index to replace the original index structure with data in the memory storage area; S3 uses a modified pre-write management system to manage multiple log files to achieve its recovery function; S4 stores the data in the memory storage area on disk and manages the generated SST files; The specific steps of S1 using the memory pool and memory pool management mechanism to maintain the data in the memory are as follows: S101 uses the memory pool to adapt according to the current amount of data in the database; S102 stores and manages the data in the memory pool through the memory storage area; The specific steps of S2 using the adaptive radix tree index to replace the original index structure with data in the memory storage area are as follows: S201 uses the data structure of the adaptive radix tree as an index of the data in the memory storage area; S202 uses the address and length of each piece of data in the memory storage area as a leaf node of the adaptive radix tree index; S203 finds the corresponding leaf node through the index, and finds the data through the leaf node; S204 uses a common database structure to adapt the adaptive radix tree; The specific steps of S3 using the modified pre-write management system to manage multiple log files and realize its recovery function are as follows: S301 manages multiple pre-written log files through component transformation, and transforms the maintenance queues of logs_ and alive_log_files_ in db_impl; S302 restores all files of the pre-write management system to the memory storage area; The specific steps of the S4 for writing the data in the memory storage area to the disk and managing the generated SST file are as follows: The data in the memory storage area of ​​S401 is sequentially written to the SST and the SST is split, and the SST is added to level 1; S402 switches the storage engine that increases memory to the rocksdb storage engine.

2. A system for increasing memory based on the rocksdb storage engine, characterized by: The system specifically includes a memory storage module, an ART index module, a pre-write management module and an SST data storage module: Memory storage module: uses memory pool and memory pool management mechanism to maintain data in memory; ART index module: uses adaptive radix tree index to replace the original index structure with data in the memory storage area; Pre-write management module: Use the modified pre-write management system to manage multiple log files and realize their recovery function; SST data storage module: It stores the data in the memory storage area on the disk and manages the generated SST files; The memory storage module specifically includes an adaptive module and a storage management module: Adaptive module: Adapts based on the current amount of data in the database through the memory pool; Storage management module: stores and manages the data in the memory pool through the memory storage area; The ART index module specifically includes an index establishment module, a node establishment module, a data index module and an index adaptation module: Index building module: using the data structure of the adaptive radix tree as the index of the data in the memory storage area; Node creation module: takes the address and length of each piece of data in the memory storage area as the leaf node of the adaptive radix tree index; Data index module: find the corresponding leaf node through the index, and find the data through the leaf node; Index adaptation module: Use the common structure of the database to adapt the adaptive radix tree; The pre-write management module specifically includes a file management module and a file recovery module: File management module: manage multiple pre-written log files through component transformation; File recovery module: restore all files in the pre-written management system to the memory storage area; The SST data storage module specifically includes a data segmentation module and an engine switching module: Data segmentation module: The data in the memory storage area is sequentially stored in the SST and the SST is segmented, and the SST is added to level 1; Engine switching module: Switch the storage engine with increased memory to the rocksdb storage engine.

Images