LSM-Tree structure storage writing performance optimization method and system

Abstract

The invention discloses an LSM-Tree structure storage and writing performance optimization method and system, and belongs to the technical field of storage and writing performance optimizing.A grouping log structure GLS is designed to be used for canceling the limitation that data on the same layer in an LSM-Tree structure are completely ordered; and respectively carrying out write operation and read operation, and maintaining the orderliness of data in a group through a grouping log structure GLS to realize write performance optimization.

Description

Technical Field

[0001] The present invention belongs to the technical field of storage write performance optimization, and in particular relates to a method and system for optimizing storage write performance of an LSM-Tree structure. Background Art

[0002] With the rapid increase in data volume and user access volume, more and more companies are choosing Key-Value databases based on the log-structured merge tree LSM-tree (Log-Structured Merge-Tree) as the underlying storage engine. Since LSM-tree can aggregate multiple random write requests into a continuous write request and write it to the external memory, LSM-tree effectively guarantees the write performance of the system. In order to ensure read performance, LSM-tree will subsequently call a series of compaction operations to sort and organize the data in the external memory. However, compaction operations will occupy a large amount of I / O resources, causing serious write amplification problems. On the one hand, the write amplification problem will cause a decrease in the overall throughput of the system; on the other hand, if the external memory uses a Solid State Drive, severe write amplification will greatly reduce its service life, which will in turn affect the reliability of the system and bring high storage costs. Therefore, reducing write amplification can save more I / O resources to serve user requests and reduce the storage cost of data.

[0003] In order to cope with the storage and query of massive data, more and more enterprises have adopted distributed database systems. Since the Key-Value database has the characteristics of simple design, high throughput and good scalability, in actual distributed database deployment, a Key-Value database is usually configured for each storage node to improve its access performance. In the entire distributed database system, the performance of each storage node will directly affect the overall performance of the system, so it is very necessary to optimize the stand-alone Key-Value database. In the Key-Value database, data is a kind of Key-L database. Compared with traditional relational databases, Key-Value databases are more advantageous in processing ultra-large-scale data. For this reason, many companies have developed exclusive Key-Value database systems, such as Google's distributed structured data storage system BigTable, Facebook's embedded key-value storage engine RocksDB and Cassandra database, Amazon's key-value storage system Dynamo, and Yahoo's server push technology Comet, etc. These Key-Value databases provide high-throughput read and write interfaces for upper-level businesses, such as cloud computing, data deduplication, online shopping, social networks and other data-intensive applications. On the one hand, Key-Value data needs to provide read and write services for upper-level applications. On the other hand, the Key-Value database needs to interact directly with the file system and even the underlying storage devices. Therefore, the design of the Key-Value database must consider both the characteristics of the upper-level applications and the characteristics of the underlying storage devices.

[0004] Since LSM-tree has good write performance, many Key-Value databases, such as Google's LevelDB and BigTable, Facebook's RocksDB and Cassandra, have adopted LSM-tree as their storage structure. The data in LSM-tree is divided into multiple layers: C0, C1, ..., C k , C k+1 , where the C0 layer is stored in memory and the other layers are stored in external memory. Data is arranged in order according to the size of the key in these layers, and the capacity of these layers increases exponentially. When a Key-Value pair is written to the LSM-tree, it is first inserted into the C0 layer in memory. When the C0 layer is full, the data in the C0 layer will be merged with the data in the C1 layer, and the merged result will be written to the C1 layer. Similarly, when the C1 layer is full, the C k The layer will also be k+1 The merged result is stored in C k+1Layers. These merge operations are called compaction operations. The purpose of compaction is to maintain the order of data in each layer according to the size of the key, so that each layer can be accessed at most once when querying the key, thereby ensuring read performance. However, compaction operations will also introduce many I / O operations, which will compete with user requests for the system's I / O resources, causing system write amplification, and thus affecting system performance.

[0005] Key-Value separation technology: As the name suggests, the Key-Value separation technology is to store the string or number Key that uniquely identifies the data and the data Value associated with the Key separately. Related work includes the Wisckey data structure and Baidu's Atlas system. Wisckey writes the Value directly into the log file Log, and encapsulates the address of the Key and Value and writes it into the LSM-tree.

[0006] Since Compaction only processes data in the LSM-tree, the Value stored in the Log will not be involved in the Compaction operation, so the system can reduce write amplification by separating the Key and the Value. However, this method will also cause some problems. First, when querying a single Key-Value pair, the system will find the location of the corresponding Value in the LSM-tree, and then read the Value from the Log. Therefore, when reading a single Key-Value pair, compared with the Key-Value overall storage method, the Key and Value separation storage method may increase additional system I / O. Secondly, when the system performs garbage collection on the Value in the Log, it needs to read the corresponding Key data in the LSM-tree. In this process, many random I / O operations will be generated, affecting system performance. In addition, the system also needs to consider the problem of data consistency in the LSM-tree and the Log, which will increase the complexity of system design. The technology of separating Key and Value can mainly solve the write amplification problem when the Value is large. In many actual workloads, the size of the Value may be only dozens of bytes or even a few bytes. The use of this technology to solve the write amplification problem does not produce a significant effect.

[0007] Pipeline compaction method: Pipeline compaction technology (PCP for short) improves the speed of compaction by exploiting the parallelism of CPU and I / O. The compaction process can be divided into three stages: reading data, sorting data, and writing back data. Among them, reading data means reading the SSTable files that need compaction from the external memory into the internal memory. Data sorting means re-sorting these SSTable files in the order of Key and then encapsulating them into new SSTable files. The final writing back of data means writing these newly generated SSTable files back to the external memory. In order to fully utilize the parallelism of CPU and I / O, PCP runs the three stages of reading data, sorting data, and writing back data in a pipeline manner. PCP uses a pipeline method to fully exploit the parallelism of CPU and I / O and improve the performance of compaction. However, PCP does not fundamentally reduce the write amplification caused by compaction, but only improves the utilization of hardware and alleviates the impact of compaction on the system. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a method and system for optimizing the storage write performance of an LSM-Tree structure in view of the deficiencies in the above-mentioned prior art, so as to solve the technical problem of write amplification caused by the Compaction mode operation occupying a large amount of I / O resources.

[0009] The present invention adopts the following technical solutions: The LSM-Tree structure storage write performance optimization method includes the following steps: Design the grouped log structure GLS to remove the restriction that the data on the same layer in the LSM-tree structure is completely ordered; Write and read operations are performed on the LSM-tree structure separately, and the grouped log structure GLS is used to maintain the order of data within the group to achieve write performance optimization.

[0010] Preferably, the group log structure GLS is specifically: The grouped log structure GLS is divided into multiple layers, including C0, C1, ..., C k , C k+1 , C0 layer is stored in memory, C1, ..., C k , C k+1 The layers are stored in the external memory, and each layer is set with m groups. The data in each group is maintained in the order of the size of the unique identifier of the data item through the Compaction mode; then the arranged data is written to the lower layer through the Compaction mode, and at this time only one unit of data is written to the lower layer.

[0011] Preferably, the Key-Value database is written only once for each layer; when all groups in a layer are full, all groups in this layer are converted into corresponding groups in the next layer through the Compaction mode.

[0012] Preferably, when the LSM-tree structure contains When there are multiple Key-Value databases, the depth of the LSM-tree structure is , Indicates the ratio of the upper limit of the number of Key-Value pairs that can be accommodated by any two adjacent layers. for: , express The upper limit of the number of Key-Value pairs that a layer can hold. express The number of Key-Value pairs that a layer can accommodate. express The number of Key-Value pairs that a layer can accommodate. express The upper limit of the number of key-value pairs that a layer can hold.

[0013] Preferably, the write operation on the LSM-tree structure is specifically as follows: When the Key-Value database is written to the group log structure GLS, it is first inserted into the group log structure GLS in memory. layer; When grouping log structure GLS When the layer is full, the grouped log structure GLS is Select an empty group in the layer from top to bottom and record this empty group as C 1j ; For grouped log structure GLS The data in the layer is sorted and written to the C 1j , when C k When there is no empty group in the layer, C k The data in the layer is loaded into memory and then sorted; Finally, the sorted data is written to C k+1 In an empty group of the layer, clear all data in the C1 layer of the grouped log structure GLS and complete the write operation on the LSM-tree structure.

[0014] Preferably, the read operation on the LSM-tree structure is specifically as follows: When searching for a specified Key in the grouped log structure GLS, start from the C0 layer and search downward layer by layer, configure the corresponding filters to access multiple groups in each layer, and the Key is a data item in the node that is used to uniquely identify the node.

[0015] In a second aspect, an embodiment of the present invention provides a system for optimizing write performance of LSM-Tree structure storage, including: Structural module, designing the grouped log structure GLS, which is used to remove the restriction that the data on the same layer in the LSM-tree structure is completely ordered; The optimization module performs write and read operations on the LSM-tree structure respectively, and maintains the order of data within the group through the group log structure GLS to achieve write performance optimization.

[0016] Preferably, the group log structure GLS is divided into multiple layers, including C0, C1, ..., C k , C k+1 , C0 layer is stored in memory, C1, ..., C k The layers are stored in external memory, and each layer is set with m groups. The data in each group is maintained in the order of the size of the unique identifier of the data item through the Compaction mode; then the arranged data is written to the lower layer through the Compaction mode, and only one unit of data is written to the lower layer at this time; When all groups in a layer are full, all groups in this layer are converted into corresponding groups in the next layer through the Compaction mode.

[0017] In a third aspect, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned LSM-Tree structure storage write performance optimization method when executing the computer program.

[0018] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned LSM-Tree structure storage write performance optimization method.

[0019] Compared with the prior art, the present invention has at least the following beneficial effects: A method for optimizing the write performance of LSM-Tree structure storage, which removes the restriction that the same-layer data in LSM-tree must be completely ordered through the grouped log structure GLS, so that when the upper layer is in compaction mode to the lower layer, it does not need to read the lower layer data. In this way, the I / O overhead caused by the compaction mode is effectively reduced, thereby reducing the write amplification of the system; in the grouped log structure GLS, when the upper layer compacts a unit of data to the lower layer, the lower layer data does not need to participate, and only this unit of data will be written to the lower layer, and the I / O overhead of the compaction mode is lower than that of LSM-tree.

[0020] Furthermore, each layer of the grouped log structure GLS is divided into several areas with non-overlapping Key ranges, and the division of areas becomes finer and finer as the number of layers increases. In the setting of the FlameDB system, the number of areas in the next layer is 10 times that of the previous layer. The Key range of the SSTable files stored in each area is included in the Key range of the area in which it is located, so the Key ranges of SSTable files in different areas of the same layer do not overlap. As user data is continuously written, the FlameDB system sorts and organizes the data through the Compaction mode to solve the problem of a single Compaction mode taking too long and occupying too much memory.

[0021] Furthermore, since there is no association between each Key-Value pair, each Key-Value pair can be stored independently in each node of the cluster. This approach makes data expansion simpler and more efficient, which is impossible with traditional relational databases.

[0022] It can be understood that the beneficial effects of the second aspect mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0023] In summary, the optimization method of the present invention can reduce I / O resource competition in the storage system and improve read and write performance.

[0024] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of the grouped log structure GLS; Figure 2 It is a schematic diagram of the process of the present invention; Figure 3 A schematic diagram of a computer device provided by an embodiment of the present invention; Figure 4The block diagram of a chip provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0028] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0029] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0030] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0031] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0032] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0033] The present invention provides a method for optimizing write performance of LSM-Tree structure storage, adopts a grouped log structure (Grouped Log-Structured, GLS), and establishes a new compaction mode on this basis. Through theoretical analysis, the grouped log structure GLS can effectively reduce write amplification; then, a KEY-VALUE based on the grouped log structure GLS is implemented, and the write speed is increased by 3 times. In terms of read performance, the read speed of the FlameDB database is only 8% slower than that of LevelDB. Under different proportions of mixed read and write workloads, the overall speed of the FlameDB database is 76% to 182% higher than that of the LevelDB database.

[0034] Example 1 See also Figure 2 The present invention provides a method for optimizing the storage write performance of an LSM-Tree structure, comprising the following steps: S1. Design the grouped log structure GLS to remove the restriction that the data at the same level in the LSM-tree must be completely ordered; Grouped log structure GLS design: See also Figure 1 , the grouped log structure GLS divides each layer of the LSM-tree structure located in the external memory into multiple groups; in the LSM-tree structure, the maximum capacity of the next layer is m times the maximum capacity of the previous layer.

[0035] The data in the grouped log structure GLS is divided into multiple layers: C0, C1, ..., C k , C k+1 , where the C0 layer is stored in memory, C1, ..., C k , C k+1 Layers are stored in external memory.

[0036] In the grouped log structure GLS, each layer is divided into m groups. The data in each group is arranged according to the size of the key. However, the grouped log structure GLS cannot guarantee that the data in the same layer is also arranged in order according to the size of the key. For this reason, multiple groups need to be accessed when querying a specified key in the same layer. Because the grouped log structure GLS cancels the restriction of complete order within the layer, when the upper layer performs the compaction mode operation on the lower layer, the system reads the data of the upper layer into the memory for sorting and then writes the sorted data to an empty group in the lower layer. The grouped log structure GLS maintains the data in each group in the order of the key through the corresponding compaction mode. Through simple observation, when a unit of data is transferred from the upper layer to the lower layer through the compaction mode in the LSM-tree, m units of data need to be read from the lower layer. When these data do not contain expired data, these m+l units of data will be written to the lower layer. In the grouped log structure GLS, when the upper layer transfers a unit of data to the lower layer through the Compaction mode, the lower layer data does not need to participate, and only this unit of data will be written to the lower layer. From the perspective of a single Key-Value pair, in the LSM-tree structure, this Key-Value pair will be written multiple times in each layer. In the grouped log structure GLS, this Key-Value pair will only be written once in each layer. In comparison, the I / O overhead of the Compaction mode in the grouped log structure GLS is significantly lower than that of the LSM-tree structure.

[0037] Each layer of the grouped log structure GLS contains m groups, and the maximum number of Key-Value pairs that can be accommodated in the C0 layer is the same as the C0 layer of the LSM-tree. When the GLS contains N Key-Value pairs, its number of layers will be the same as the LSM-tree, that is, , Indicates the ratio of the upper limit of the number of Key-Value pairs that can be accommodated by any two adjacent layers. for: , express The upper limit of the number of Key-Value pairs that a layer can hold. express The number of Key-Value pairs that a layer can accommodate. express The number of Key-Value pairs that a layer can accommodate. express The upper limit of the number of key-value pairs that a layer can hold.

[0038] In the grouped log structure GLS, when all groups in a layer are full, the compaction mode converts all groups in this layer into corresponding groups in the next layer. Therefore, in the grouped log structure GLS, each Key-Value pair will only be written once in each layer. Similar to LSM-tree, most of the Key-Value data in the grouped log structure GLS will also pass through each layer and finally be written to the last layer. Then the write amplification of the grouped log structure GLS is: -1 After analysis, it can be seen that the write amplification of the grouped log structure GLS is only 2 / m of the write amplification of the LSM-tree.

[0039] S2, write operation, the group log structure GLS maintains the order of data within the group; Write process: When a Key-Value pair is written to the grouped log structure GLS, it is first inserted into the memory layer; when When the layer is full, the grouped log structure GLS selects an empty group from the C1 layer in order from top to bottom and records this empty group as C 1j ; Then group the log-structured GLS pairs Sort the data in the layer and write the sorted data into the empty group C 1j , similarly when C k When there is no empty group in the layer, the system will C k The data in the layer is loaded into memory and then sorted; Finally, the sorted data will be written to C k+1 All data in the original C1 layer will be cleared. The grouped log structure GLS maintains the order of data in the group in this way.

[0040] S3. Read operation.

[0041] Reading process: When searching for a specified key in the grouped log structure GLS, the system starts from the C0 layer and searches downward layer by layer. However, unlike LSM-tree, the grouped log structure GLS contains multiple groups in each layer. Therefore, when querying data, GLS needs to access multiple groups in each layer. From this point of view, the read performance of the grouped log structure GLS is worse than that of LSM-tree. In order to improve the read performance of the grouped log structure GLS, configure the corresponding filter for the grouped log structure GLS to avoid unnecessary external memory access.

[0042] Example 2 The present invention provides an LSM-Tree structure storage write performance optimization system, which can be used to implement the above-mentioned LSM-Tree structure storage write performance optimization method. Specifically, the LSM-Tree structure storage write performance optimization system includes a GLS module and an optimization module.

[0043] Among them, the structural module designs the grouped log structure GLS to remove the restriction that the data at the same level in the LSM-tree structure is completely ordered; The grouped log structure GLS is divided into multiple layers, including C0, C1, ..., C k , C k+1 , C0 layer is stored in memory, C1, ..., C k , C k+1 The layers are stored in the external memory, and each layer is set with m groups. The data in each group is maintained in the order of the size of the unique identifier of the data item through the Compaction mode; then the arranged data is written to the lower layer through the Compaction mode, and only one unit of data is written to the lower layer at this time; When all groups in a layer are full, all groups in this layer are converted into corresponding groups in the next layer through the Compaction mode.

[0044] The optimization module performs write and read operations on the LSM-tree structure respectively, and maintains the order of data within the group through the group log structure GLS to achieve write performance optimization.

[0045] Example 3 The present invention provides a terminal device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, graphics processors (GPU), tensor processors (TPU), digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the LSM-Tree structure storage write performance optimization method, including: A grouped log structure GLS is designed to remove the restriction that data on the same layer in the LSM-tree structure is completely ordered. Write and read operations are performed on the LSM-tree structure separately, and the grouped log structure GLS is used to maintain the order of data within the group, thereby optimizing write performance.

[0046] See also Figure 3 , the terminal device is a computer device, and the computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the * method in the embodiment is implemented. To avoid repetition, it is not described here one by one. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the * system of the embodiment are implemented. To avoid repetition, it is not described here one by one.

[0047] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Figure 3 This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0048] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, graphics processing units (GPU), tensor processing units (TPU), digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0049] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 60.

[0050] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0051] See also Figure 4 The terminal device is an electronic device 600, which is in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0052] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 2 Follow the steps shown in .

[0053] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0054] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0055] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0056] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems). Such communication may be performed via an input / output interface 650. In addition, the electronic device 600 may also communicate with one or more networks (e.g., local area networks, wide area networks, and / or public networks, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0057] Example 4 The present invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device, and can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that more specific examples of the computer-readable storage medium here include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0058] Computer readable storage media also include data signals propagated in baseband or as part of a carrier wave, which carry readable program code. Such propagated data signals can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than a readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency, etc., or any suitable combination of the above.

[0059] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network or a wide area network, or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0060] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the LSM-Tree structure storage write performance optimization method in the above embodiment; the processor may load and execute the following steps: A grouped log structure GLS is designed to remove the restriction that data on the same layer in the LSM-tree structure is completely ordered. Write and read operations are performed on the LSM-tree structure separately, and the grouped log structure GLS is used to maintain the order of data within the group, thereby optimizing write performance.

[0061] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0063] The performance of the FlameDB database was tested. The compaction overhead of the FlameDB database and the LevelDB database, as well as the read and write performance of the two were mainly compared. The FlameDB database and the LevelDB database run on a Linux server.

[0064] Write performance test The workload of the test is generated by the performance testing tool YCSB. 50GB of Key-Value data is generated by YCSB, where each Key-Value pair is 1KB in size. These Key-Value data are then written to the FlameDB database and the LevelDB database respectively. In the experiment, the read and write I / O volume of the FlameDB database and the LevelDB database were recorded, and their write amplification was calculated using the above data. In addition, the write throughput of the FlameDB database and the LevelDB database was also measured. In order to observe the impact of external memory on system performance, Solid State Drive SSD and Hard Disk Drive HDD were selected as the storage media for external memory.

[0065] The amount of read and write I / O caused by the Compaction mode in the FlameDB database and the LevelDB database. It was found that the amount of read I / O caused by the Compaction mode in the FlameDB database and the LevelDB database increased as the amount of Key-Value data written increased. When writing 50GB of Key-Value data to the LevelDB database, the Compaction mode caused 750GB of read I / O, while in the FlameDB database, the system's read I / 0 caused by the Compaction mode was only 185GB. Compared with the LevelDB database, the amount of read I / O caused by the Compaction mode in the FlameDB database is only 1 / 4 of the former. Because the write load does not include deletion and update operations, the amount of read I / O and write I / O caused by the Compaction mode in the FlameDB database and the LevelDB database are basically the same. It should be noted that the amount of read and write I / O caused by Compaction is unrelated to the underlying storage medium.

[0066] After recording the read and write I / O volume of compaction in FlameDB and LevelDB, the write amplification of the two systems was calculated. It can be observed that as the amount of data written increases, the system write amplification of FlameDB and LevelDB also increases. This is because when the amount of data written increases, the number of layers of FlameDB and LevelDB will increase, which is the reason for the increase in write amplification. Compared with LevelDB, FlameDB has a smaller database write amplification, which means that more I / O resources can be used to serve user requests, so the write throughput of FlameDB is much higher than that of LevelDB. Whether on SSD or HDD, the write throughput of FlameDB is 3 times higher than that of LevelDB on average. In addition, it is also found that for FlameDB and LevelDB, the write throughput on SSD is only 1.3 times that on HDD. There are two main reasons for this phenomenon. First, compared with random read and write, the performance of HDD in continuous read and write is not much different from that of SSD. Secondly, when a large amount of data is written to the SSD, the SSD may perform garbage collection operations, which will seriously affect the write performance of the system.

[0067] Read performance test The read performance of the FlameDB database and the LevelDB database was tested. In many practical applications, the storage system usually improves access performance by loading metadata into memory. This is because metadata takes up much less space than the data itself. First, 50GB of Key-Value data was written into the FlameDB database and the LevelDB database respectively, and then the system loaded all the index information and Bloom filters of the Key-Value data into memory. Then YCSB was used to generate 100,000 random read request operations, and the keys accessed by these read requests were evenly distributed. In the experiment, unbuffered IO (DirectIO) was used to bypass the file system's page cache and read Key-Value data directly from external memory.

[0068] In order to compare the impact of Bloom filter size on read performance, different space configurations were set for Bloom filter in the experiment. Under these four configurations, Bloom filter allocated 4 bits, 8 bits, 12 bits and 16 bits of memory space for each key respectively. When Bloom filter allocated the same size of space for each key, the read throughput of LevelDB database was higher than that of FlameDB database, because the query range of LevelDB database was smaller than that of FlameDB database. It can also be observed that when Bloom filter allocated more space for each key, the read throughput of LevelDB database and FlameDB database increased. This is because more Bloom filter space is allocated to each key, which will reduce the false alarm rate of Bloom filter, thereby helping FlameDB database and LevelDB database to reduce the number of read I / 0 during query, thus correspondingly improving the read throughput. In addition, it was found that as the space allocated by Bloom filter for each key increased, although the read throughput of FlameDB database and LevelDB database increased, the growth rate slowed down to varying degrees. When the space allocated by the Bloom filter for each key increases from 4 bits to 8 bits, the read throughput of the LevelDB database increases by 67%, and the read throughput of the FlameDB database increases by 108%. When the space allocated by the Bloom filter for each key increases from 12 bits to 16 bits, the read throughput of the LevelDB database increases by less than 3%, and the read throughput of the FlameDB database increases by 16%. There are two reasons for this. First, when the space allocated by the Bloom filter for each key reaches a certain size, its false alarm rate is already very small. Even if the space is increased, the false alarm rate will not be significantly reduced. Therefore, when the space allocated by the Bloom filter for the key increases from 12 bits to 16 bits, the read throughput of the LevelDB database only increases slightly. Compared with the LevelDB database, the query range of the FlameDB database is larger, so the FlameDB database is more sensitive to the false alarm rate. When the false alarm rate of the Bloom filter decreases, the FlameDB database can more effectively narrow the query range and reduce the number of read I / Os caused by the query. As the space allocated by the Bloom filter for each key increases, the gap between the read throughput of the FlameDB database and the LevelDB database is narrowing. When the Bloom filter allocates 2 bits of storage space for each key, the read throughput of the FlameDB database is almost half of that of the LevelDB database. When the Bloom filter allocates 16 bits of space for each key, the gap between the read throughput of the FlameDB database and the LevelDB database is only 8%.Compared with the significant improvement in write performance, the slight decrease in read performance of the FlameDB database is acceptable.

[0069] The same experiment was also conducted on HDD. The read throughput of LevelDB database is still higher than that of FlameDB database, and as the Bloom filter allocates more space for each key, the read throughput of both databases gradually increases. In addition, it can be observed that the read performance of FlameDB database and LevelDB database on SSD is about 24 times faster than that on HDD. In the previous experiment, it was found that the difference in write speed between the two systems on SSD and HDD is not very large. This is because both FlameDB database and LevelDB database accumulate user random write requests to a certain extent and then write them to the external memory in batches, so that random writes can be converted into continuous writes. For random reads, the location of data stored in the external memory is fixed, and the system needs to read the data from the specified location, which will cause a lot of random I / O. Compared with the performance gap between SSD and HDD in continuous read and write, the performance of SSD in random read and write is much higher than that of HDD. For this reason, many companies improve the random read performance of the system by replacing the external memory with SSD.

[0070] In summary, the present invention provides a method and system for optimizing the storage write performance of an LSM-Tree structure, which can reduce the overhead of compaction operations.

[0071] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0072] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0073] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0074] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0075] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0076] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0077] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0078] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0079] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0081] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. LSM-Tree structure storage write performance optimization method, characterized in that: The following steps are involved: Design the grouped log structure GLS to remove the restriction that data on the same layer is completely ordered in the LSM-tree structure; Write and read operations are performed on the LSM-tree structure separately, and the group log structure GLS is used to maintain the order of data within the group to achieve write performance optimization.

2. The LSM-Tree structure storage write performance optimization method according to claim 1, characterized in that: The grouped log structure GLS is as follows: The grouped log structure GLS is divided into multiple layers, including C0, C1, ..., C k , C k+1 , C0 layer is stored in memory, C1, ..., C k , C k+1 The layers are stored in the external memory, and each layer is set with m groups. The data in each group is maintained in the order of the size of the unique identifier of the data item through the Compaction mode; then the arranged data is written to the lower layer through the Compaction mode, and at this time only one unit of data is written to the lower layer.

3. The LSM-Tree structure storage write performance optimization method according to claim 2, characterized in that: The Key-Value database writes each layer only once; when all groups in a layer are full, all groups in this layer are converted into corresponding groups in the next layer through the Compaction mode.

4. The LSM-Tree structure storage write performance optimization method according to claim 1, characterized in that: When the LSM-tree structure contains When there are multiple Key-Value databases, the depth of the LSM-tree structure is , Indicates the ratio of the upper limit of the number of Key-Value pairs that can be accommodated by any two adjacent layers. for: , express The upper limit of the number of Key-Value pairs that a layer can hold. express The number of Key-Value pairs that a layer can accommodate. express The number of Key-Value pairs that a layer can accommodate. express The upper limit of the number of key-value pairs that a layer can hold.

5. The LSM-Tree structure storage write performance optimization method according to claim 1, characterized in that: The specific write operations on the LSM-tree structure are: When the Key-Value database is written to the group log structure GLS, it is first inserted into the group log structure GLS in memory. layer; When grouping log structure GLS When the layer is full, the grouped log structure GLS is Select an empty group in the layer from top to bottom and record this empty group as C 1j ; For grouped log structure GLS The data in the layer is sorted and written to the C 1j , when C k When there is no empty group in the layer, C k The data in the layer is loaded into memory and then sorted; Finally, the sorted data is written to C k+1 In an empty group of the layer, clear all data in the C1 layer of the grouped log structure GLS and complete the write operation on the LSM-tree structure.

6. The LSM-Tree structure storage write performance optimization method according to claim 1, characterized in that: The specific read operations on the LSM-tree structure are: When searching for a specified Key in the grouped log structure GLS, start from the C0 layer and search downward layer by layer, configure the corresponding filters to access multiple groups in each layer, and the Key is a data item in the node that is used to uniquely identify the node.

7. A LSM-Tree structure storage write performance optimization system, characterized in that: include: Structural module, designing the grouped log structure GLS, which is used to remove the restriction that the data on the same layer in the LSM-tree structure is completely ordered; The optimization module performs write and read operations on the LSM-tree structure respectively, and maintains the order of data within the group through the group log structure GLS to achieve write performance optimization.

8. The LSM-Tree structure storage write performance optimization system according to claim 7, characterized in that: The grouped log structure GLS is divided into multiple layers, including C0, C1, ..., C k , C k+1 , C0 layer is stored in memory, C1, ..., C k The layers are stored in external memory, and each layer is set with m groups. The data in each group is maintained in the order of the size of the unique identifier of the data item through the Compaction mode; then the arranged data is written to the lower layer through the Compaction mode, and only one unit of data is written to the lower layer at this time; When all groups in a layer are full, all groups in this layer are converted into corresponding groups in the next layer through the Compaction mode.

9. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 6.

10. A computing device, characterized in that include: One or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include steps for executing the method described in any one of claims 1 to 6.

Images