A method and system for compression operation of LSM tree storage engine

Abstract

The present application relates to a kind of compression operation method and system for LSM tree storage engine, including configuration minimum variable and maximum variable and according to the result of file intersection check, different files with intersection are merged, by special design to the size limit of FileTable file, it is allowed in a certain range, unnecessary file can be avoided to be read and written to meet the size hard requirement of FileTable file, when executing FileTable merge, avoid unnecessary file to be read and written due to the size of non-end file of FileTable is close to or equal to MaxFileSize, reduce disk IO, so as to improve overall write and query performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database operation and data storage, and particularly relates to a compression operation method and system for an LSM tree storage engine. BACKGROUND

[0002] The storage engine based on the LSM tree (Log-Structured Merge-Tree) and its variants (hereinafter referred to as the LSM tree storage engine) solves the write performance bottleneck problem caused by random writing of the traditional storage engine based on the B+ tree and its variants (hereinafter referred to as the B+ tree storage engine).

[0003] The LSM tree storage engine is optimized for writing, and the basic data storage processing idea thereof includes: (1) a write request writes data to a disk log file WAL (Write Ahead Log) and to a memory MemTable. The WAL realizes data persistence, and ensures that data is not lost in the event of system exceptions, but the data is disordered. The MemTable is a copy of the WAL, and the data is ordered, facilitating quick query of the data of the WAL. (2) When the size of the MemTable reaches a certain threshold, or the time interval from the last Compaction reaches a certain threshold, the storage engine triggers the execution of a Compaction. (3) The role of the compression operation (Compaction) is to optimize query. As more and more data is written into the storage engine, the query performance becomes worse due to the large number of MemTable / WALs, the existence of duplicate data, the overall disorder of data, and the lack of index. The Compaction combines multiple WALs into a FileTable, and combines multiple FileTables into a FileTable. By reducing the number of WALs and FileTables, and ensuring the order of data, the query process can greatly reduce the number of files that need to be scanned, thereby improving the query performance.

[0004] The Compaction process of the existing storage engine combines multiple FileTables into a FileTable after selecting the FileTable, using a merge sort algorithm. In this process, in order to ensure that the size of the FileTable file meets certain restrictions, most FileTable files need to be completely read and rewritten, resulting in unnecessary disk IO, and further affecting the overall write and query performance. SUMMARY

[0005] In order to solve the problems in the prior art, the present application provides a compression operation method and system for an LSM tree storage engine, wherein the size limit of a FileTable file is specially designed, so that unnecessary files can be avoided within a certain range to meet the size hard requirement of the FileTable file, and unnecessary files are avoided to be read and written when the FileTable is merged due to the size of the non-end file of the FileTable being close to or equal to MaxFileSize, so as to reduce disk IO and improve the overall write and query performance.

[0006] In order to achieve the above object, the technical scheme adopted by the present application comprises:

[0007] A compression operation method for an LSM tree storage engine, characterized in that it comprises:

[0008] S1, creating a first list and a second list, and adding all files to be operated into the first list;

[0009] S2, configuring a minimum variable and a maximum variable;

[0010] S3, checking whether the first list is empty, and ending the compression operation when the first list is empty;

[0011] S4, when the first list is not empty, transferring the file ranked first in the current first list into the second list, and assigning the start value and the end value of the file to the minimum value and the maximum value to generate a first numerical interval;

[0012] S5, checking whether the first list is empty, and executing step S9 when the first list is empty, or executing step S6 when the first list is not empty;

[0013] S6, recording the start value and the end value of the file ranked first in the current first list as a second numerical interval, checking whether there is an intersection between the second numerical interval and the first numerical interval, and executing step S7 when there is an intersection, or executing step S9 when there is no intersection;

[0014] S7, calculating the union set between the second numerical interval and the first numerical interval to obtain a third numerical interval, and reassigning the minimum value and the maximum value using the third numerical interval to generate a new first numerical interval;

[0015] S8, transferring the file ranked first in the current first list into the second list, and returning to execute step S5 again;

[0016] S9, checking whether there is only one file in the second list;

[0017] S10, when there is only one file in the second list, creating a file hard link corresponding to the file, and returning to execute step S3 again;

[0018] S11、when there are multiple files in the second list, merging the multiple files in the second list, and returning to re-execute step S3.

[0019] Further, the merging the multiple files in the second list comprises:

[0020] configuring a file volume fluctuation amplitude parameter a, the file fluctuation amplitude parameter a has a value range of 0% to 100%;

[0021] obtaining a preset file volume threshold M of the LSM tree storage engine;

[0022] using formula 1 and formula 2 to respectively calculate the lower limit R1 and the upper limit R2 of the file volume range interval according to the preset file volume threshold M and the file volume fluctuation amplitude parameter a;

[0023]

[0024] R2=(1+a)×M formula 2

[0025] merging the multiple files in the second list into one or two new files in the file volume range interval according to the file volume range interval.

[0026] The application also relates to a compression operation system for an LSM tree storage engine, characterized in that the system comprises:

[0027] a variable control module configured to configure and assign values to a minimum value variable and a maximum value variable;

[0028] a first checking module configured to check whether the first list is empty;

[0029] a second checking module configured to check whether the second list is empty;

[0030] a first execution module configured to transfer files from the first list to the second list;

[0031] a second execution module configured to create file hard links;

[0032] a third execution module configured to merge multiple files.

[0033] The application also relates to a computer readable storage medium, characterized in that the storage medium stores a computer program, and the computer program is executed by a processor to implement the above method.

[0034] The application also relates to an electronic device, characterized in that the device comprises a processor and a memory;

[0035] the memory is configured to store the first list and the second list;

[0036] The processor is configured to execute the above method by calling the first list and the second list.

[0037] The present application also relates to a computer program product comprising computer programs and / or instructions, characterized in that the computer programs and / or instructions, when executed by a processor, implement the steps of the above method.

[0038] The present application has the following beneficial effects:

[0039] The method and system for compression operation of an LSM tree storage engine according to the present application can avoid reading and writing unnecessary files in a certain range by specially designing the size limit of a FileTable file, can avoid reading and writing unnecessary files to meet the hard requirement of the size of the FileTable file during FileTable merging, and can reduce disk IO, thereby improving overall write and query performance. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The figure is a flowchart of the method for compression operation of an LSM tree storage engine according to the present application.

[0041] Figure 2 The figure is a structural diagram of the system for compression operation of an LSM tree storage engine according to the present application.

[0042] Figure 3 The figure is a schematic diagram of the prior art compression operation process.

[0043] Figure 4 The figure is a schematic diagram of the compression operation process of the preferred embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to more clearly understand the content of the present application, the present application will be described in detail in combination with the drawings and embodiments.

[0045] The first aspect of the present application relates to a method for compression operation of an LSM tree storage engine, as shown in the step flow Figure 1 The method comprises the following steps:

[0046] S1, creating a first list and a second list, and adding all files to be operated to the first list;

[0047] S2, configuring a minimum variable and a maximum variable;

[0048] S3, checking whether the first list is empty, and ending the compression operation when the first list is empty;

[0049] S4, when the first list is not empty, transferring the file ranked first in the current first list into the second list, and assigning the start value and end value of the file to the minimum value and maximum value, generating a first numerical interval;

[0050] S5, checking whether the first list is empty, when the first list is empty, executing step S9, and when the first list is not empty, executing step S6;

[0051] S6, recording the start value and end value of the file ranked first in the current first list as a second numerical interval, and checking whether there is an intersection between the second numerical interval and the first numerical interval, when there is an intersection, executing step S7, and when there is no intersection, executing step S9;

[0052] S7, calculating the union set between the second numerical interval and the first numerical interval to obtain a third numerical interval, and reassigning the minimum value and maximum value using the third numerical interval to generate a new first numerical interval;

[0053] S8, transferring the file ranked first in the current first list into the second list, and returning to re-execute step S5;

[0054] S9, checking whether there is only one file in the second list;

[0055] S10, when there is only one file in the second list, creating a file hard link corresponding to the file, and returning to re-execute step S3;

[0056] S11, when there are multiple files in the second list, merging the multiple files in the second list, and returning to re-execute step S3. For the merging of multiple files, preferably based on a merge sort algorithm, including: configuring a file size fluctuation amplitude parameter a, the value range of the file size fluctuation amplitude parameter a being 0% to 100%; obtaining a preset file size threshold M of an LSM tree storage engine; using formula 1 and formula 2 to calculate the lower limit R1 and the upper limit R2 of the file size range interval according to the preset file size threshold M and the file size fluctuation amplitude parameter a;

[0057]

[0058] R2 = (1 + a) x M Formula 2

[0059] According to the file size range interval, the multiple files in the second list are merged into one or two new files within the file size range interval.

[0060] By performing the above operation, one FileTable non-end file can be merged with any FileTable file to generate one or two files with sizes within the above file size range interval (which can be used as non-end files). This is the key to reducing file read and write operations: as long as non-end files are involved, only files with intersections within the third value interval need to be processed, and other unrelated files do not need to be processed.

[0061] The above method is further illustrated by a specific embodiment.

[0062] As Figure 3 shown is a process diagram of a compression operation in the prior art, which requires all files of FileTable A, FileTable B and FileTable C to be converted into FileTable D after compression operation. As Figure 3 shown, in the prior art, only the non-end file of FileTable A needs to be generated as a new file by creating a file hard link, and other files need to be completely read and rewritten (the purpose is to maintain the size of the non-end file of FileTable D close to or equal to MaxFileSize).

[0063] As Figure 4 shown is a process diagram of the execution of the preferred embodiment of the compression operation method of the present application in the same application scenario. Compared with Figure 3 the prior art, the preferred embodiment only needs to read the last file of FileTable A, the first and last files of FileTable B, and the first file of FileTable C, and other files are completed by creating a file hard link. Among them, assuming that FileTable A, FileTable B and FileTable C all have 10 files, the size of the non-end file is S, and the size of the last file is 0.2S, then the disk IO read amount of the preferred embodiment is 2.4S (0.2S+1S+0.2S+1S), which is more than 80% lower than the disk IO read amount of the prior art of 18.6S (0.2S+9S+0.2S+9S+0.2S), and the more the number of files in actual application, the greater the advantage of using the method of the present application for compression operation.

[0064] Another aspect of the present application also relates to a compression operation system for an LSM tree storage engine, which structure is as Figure 2 shown, comprising:

[0065] a variable control module for configuring and assigning values to the minimum value variable and the maximum value variable;

[0066] a first checking module for checking whether the first list is empty;

[0067] The second checking module is configured to check whether the second list is empty.

[0068] The first executing module is configured to transfer the files from the first list to the second list.

[0069] The second executing module is configured to create a hard link of the files.

[0070] The third executing module is configured to merge the files.

[0071] By using the system, the above-mentioned operation processing method can be executed and the corresponding technical effects can be achieved.

[0072] The embodiment of the present application further provides a computer readable storage medium capable of realizing all steps of the method in the above-mentioned embodiment, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize all steps of the method in the above-mentioned embodiment.

[0073] The embodiment of the present application further provides an electronic device for executing the above-mentioned method, and the electronic device at least has a processor and a memory, and in particular, the memory stores data and related computer programs required for executing the method, for example, the first list and the second list, and all steps of the method are realized by calling the data and programs in the memory by the processor, and the corresponding technical effects are obtained.

[0074] Preferably, the electronic device can include a bus architecture, and the bus can include any number of interconnected buses and bridges, and the bus links various circuits including one or more processors and memories together. The bus can also link various other circuits such as peripheral devices, voltage stabilizers and power management circuits, which are well known in the art, and thus, further description thereof will not be given herein. The bus interface provides an interface between the bus and the receiver and the transmitter. The receiver and the transmitter can be the same element, i.e., a transceiver, which provides a unit for communicating with various other systems on a transmission medium. The processor is responsible for managing the bus and general processing, and the memory can be used to store data used by the processor when performing operations.

[0075] Additionally, the electronic device can further include a communication module, an input unit, an audio processor, a display, a power supply, etc. The processor (or controller, operating control) employed can include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of the various components of the electronic device; the memory can be one or more of a cache, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device, which stores the above-mentioned data information, and further stores programs for executing the relevant information, and the processor can execute the programs stored in the memory to achieve information storage or processing, etc.; the input unit is used to provide input to the processor, and can be a key or touch input device, for example; the power supply is used to provide power to the electronic device; the display is used to display display objects such as images and text, and can be an LCD display, for example. The communication module is a transmitter / receiver that transmits and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, which can be the same as in a conventional mobile communication terminal. Based on different communication technologies, multiple communication modules can be provided in the same electronic device, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module, etc. The communication module (transmitter / receiver) is also coupled to the speaker and the microphone via the audio processor to provide audio output via the speaker and receive audio input from the microphone, thereby achieving the usual telecommunication functions. The audio processor can include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor is also coupled to the central processor, so that it can record on the local machine through the microphone, and it can play the stored sound on the local machine through the speaker.

[0076] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0077] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0078] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0079] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0080] The above description is only preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed by the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for compression operation of an LSM tree storage engine, characterized in that, Comprising: S1, creating a first list and a second list, and adding all files to be operated into the first list; S2, configuring a minimum value variable and a maximum value variable; S3, checking whether the first list is empty, and ending the compression operation when the first list is empty; S4, when the first list is not empty, transferring the file ranked first in the current first list into the second list, and assigning the start value and the end value of the file to the minimum value and the maximum value to generate a first numerical interval; S5, checking whether the first list is empty, and executing step S9 when the first list is empty, or executing step S6 when the first list is not empty; S6, recording the start value and the end value of the file ranked first in the current first list as a second numerical interval, checking whether there is an intersection between the second numerical interval and the first numerical interval, and executing step S7 when there is an intersection, or executing step S9 when there is no intersection; S7, calculating the union set between the second numerical interval and the first numerical interval to obtain a third numerical interval, and reassigning the minimum value and the maximum value using the third numerical interval to generate a new first numerical interval; S8, transferring the file ranked first in the current first list into the second list, and returning to execute step S5 again; S9, checking whether there is only one file in the second list; S10, when there is only one file in the second list, creating a file hard link corresponding to the file, and returning to execute step S3 again; S11, when there are multiple files in the second list, merging the multiple files in the second list, and returning to execute step S3 again; The merging of the multiple files in the second list comprises: configuring a file size fluctuation amplitude parameter a, the value range of the file size fluctuation amplitude parameter a being 0% to 100%; obtaining a preset file size threshold M of an LSM tree storage engine; calculating the lower limit R1 and the upper limit R2 of the file size range interval according to the preset file size threshold M and the file size fluctuation amplitude parameter a using formula 1 and formula 2 respectively; R2=(1+a)×M Formula 2 merging the multiple files in the second list into one or two new files within the file size range interval according to the file size range interval, and generating one or two files with sizes conforming to the above file size range interval range after merging one FileTable non-end file with any one FileTable file.

2. A compression operating system for an LSM tree storage engine, the compression operating system comprising: For executing the method of claim 1, comprising: a variable control module for configuring a minimum value variable and a maximum value variable and performing corresponding assignment; a first checking module for checking whether the first list is empty; a second checking module for checking whether the second list is empty; a first execution module for transferring files from the first list into the second list; a second execution module for creating a file hard link; a third execution module for merging multiple files.

3. A computer-readable storage medium, characterized in that, The storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the method of claim 1.

4. An electronic device, comprising: Comprising a processor and a memory; The memory is used for storing a first list and a second list; The processor is configured to execute the method of claim 1 by invoking the first list and the second list.

5. A computer program product comprising computer programs and / or instructions, characterized in that, The computer program and / or instructions, when executed by a processor, implement the steps of the method of claim 1.

Images