A key-value storage method and system based on an LSM tree

Abstract

The application discloses a key value storage method and system based on an LSM tree, comprising three parts of a storage structure in a memory, a log file in a hard disk and a data file in the hard disk, the storage structure in the memory is used as a cache to support efficient access of data, the log file in the hard disk is responsible for two functions of fault recovery and data persistence, and the data file in the hard disk dynamically adjusts the layout of hot and cold data by means of a merging process, through cooperation of the three parts, differential treatment can be performed on the hot and cold data, the access performance of the LSM tree is improved, and the problems of read amplification and read performance imbalance of the LSM tree can be relieved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data storage, and particularly relates to a key value storage method and system based on an LSM tree. BACKGROUND

[0002] As relational databases face great challenges in data storage applications in the big data era, people begin to explore new generation database technologies to make up for the shortcomings of relational databases, so that the new generation database technologies can play a better role in the big data era, and NoSQL databases emerge as the times require. The LSM tree is a kind of efficient storage structure, and many NoSQL databases are implemented based on the LSM tree as a prototype, such as LevelDB, RocksDB and the like.

[0003] The main design idea of the LSM tree is to transform as many random IOs as possible into sequential IOs to obtain higher read-write performance, by taking advantage of the characteristic that the sequential IO performance of a hard disk is much higher than the random IO performance. When data is written, the data is organized in an ordered form and written into the hard disk, and when certain conditions are met, the data in different levels of data files is read sequentially, and the data is sorted and then sequentially written into a new data file in a specific level. When data is read, the files in different levels are read sequentially, and the index, Bloom filter and other structures are used to obtain better query performance.

[0004] In the scenario where the access of hot data and cold data is quite different, generally speaking, only when the hot data has good access performance and the cold data has poor access performance, can better access performance be obtained as a whole. The traditional LSM tree does not differentiate the hot data and the cold data, and how to make the LSM tree obtain better access performance in the scenario where the access of hot data and cold data is quite different is one of the important technical problems in the field. SUMMARY

[0005] The technical problem to be solved by the application is to provide a key value storage method and system based on an LSM tree, to solve the technical problem that hot data and cold data are stored in the LSM tree without differentiation, and the storage performance of the LSM tree cannot be released in the scenario where the access of hot data and cold data is quite different.

[0006] The application adopts the following technical scheme:

[0007] A key value storage method based on an LSM tree comprises the following steps:

[0008] S1, hot data is stored in a memory space, and cold data is evicted.

[0009] S2, manage the log file in the hard disk, write in the integer multiple of the hard disk block as the unit, read through the index in the memory to speed up the reading speed;

[0010] S3, based on the multi-level multi-file structure, arrange the data file in the hard disk through the merging operation, realize the key value storage based on the LSM tree.

[0011] Specifically, step S1 is specifically:

[0012] S101, open space in the memory to accommodate user data;

[0013] S102, count the information of the data in the storage structure, the information of the data in the storage structure includes one or more of the access frequency, access time and semantic relationship between data;

[0014] S103, sort the data according to the information counted in step S102, determine the heat of the data according to the result of sorting, and divide the data into hot data or cold data;

[0015] S104, when the capacity of the memory storage structure is full, delete one or more data in the memory storage structure;

[0016] S105, after step S104 eliminates the cold data, all the statistical information of the remaining data is retained or all the statistical information of the remaining data is cleared or periodically cleared.

[0017] Further, in step S101, the storage structure used for storing data includes one or more of linked list, hash table and tree.

[0018] Further, in step S102, for the encapsulated data saved in the memory, the reference count of the encapsulated data object is counted.

[0019] Specifically, step S2 is specifically:

[0020] S201, temporarily store data in the memory, the temporarily stored data includes the key, value field of user data, the type of operation;

[0021] S202, when the size of the data in the memory reaches the preset threshold, serialize the data and write it into the log file in the unit of hard disk block, and update the index of the corresponding data in the memory, the index includes the key field and the value field of the data in the log file;

[0022] S203, when the size of the log file reaches the preset threshold, first write the remaining data in the memory into the log file, and then write the index of the data to the end of the log file to form a complete log file;

[0023] S204, the log file is handed over to the data file part in the hard disk for management, and a new file is recreated as a log file to receive the writing of the data stream.

[0024] Further, in step S204, the data file in the hard disk triggers the merging operation when the condition is met, which is divided into:

[0025] The first type of merging, that is, the files participating in the merging operation are the data files of level 0 and level 1 layers;

[0026] The second type of merging, that is, the files participating in the merging operation are the data files of level i (i>0) and level i+1 layers.

[0027] Further, the trigger condition of the merging operation is:

[0028] The number of data files in a certain layer reaches a preset threshold; and / or, the total size of data files in a certain layer reaches a preset threshold; and / or, the total number of invalid reads of a certain layer file reaches a preset threshold; and / or, the space amplification of a certain layer file reaches a preset threshold.

[0029] Specifically, step S3 is specifically:

[0030] S301, selection of the files to be merged, the selection strategy includes: polling selection, selection of files with small overlapping ranges, selection of files with low heat, and selection of files with more deletion / update marks;

[0031] S302, reading the data files of the adjacent two layers, rearranging the data in them to generate a new file, and placing the new file to the corresponding layer;

[0032] S303, retaining the new file generated after merging and cleaning the invalid old files.

[0033] Further, in step S302, for the first type of merging, the files of level 0 layer and level 1 layer are read when executed, the data in them is rearranged to generate a new file, and the generated new file is placed to level 1 layer;

[0034] For the second type of merging, the files of level i layer and level i+1 layer are read when executed, i>0, the data in them is rearranged to generate a new file, and the file containing hot data is placed to level i layer and the file containing cold data is placed to level i+1 layer.

[0035] In a second aspect, an embodiment of the present application provides a key-value storage system based on an LSM tree, comprising:

[0036] A memory module uses memory space to store hot data and evict cold data;

[0037] A log module manages a log file in the hard disk, writes in an integer multiple of a hard disk block as a unit, and reads through an index in the memory to speed up the reading speed;

[0038] A data module arranges a data file in the hard disk through a merging operation based on a multi-level multi-file structure, and realizes key value storage based on an LSM tree.

[0039] Compared with the prior art, the present application has at least the following beneficial effects:

[0040] A key value storage method based on an LSM tree simplifies the writing process of the LSM tree by redesigning the management modes of the memory space, the log file and the data file, differentiates hot data and cold data, makes the data reading process of the LSM tree more suitable for application scenarios with large differences in hot data and cold data access, and thus improves the read-write performance of the LSM tree, relieves the read amplification problem, and improves the balance of the system.

[0041] Further, step S1 differentiates and converts hot data and cold data using the memory space, and expels cold data when the space is full, effectively manages the memory space, and makes the most frequently accessed data stay in the memory for a long time to obtain relatively good access performance, thereby improving the overall data access efficiency.

[0042] Further, for data storage, different data structures have different advantages and disadvantages, and the space efficiency corresponding to different data structures is different, and the performance of insertion, deletion, update and search operations is also different. In the memory, the storage structure used to store data is usually one or a combination of a linked list, a hash table and a tree. The data structure is selected flexibly to balance time efficiency and space efficiency.

[0043] Further, some data content is relatively complex and can be divided into multiple parts, and the access situation of each part also differs. Such data is encapsulated and then saved to the memory. For the data saved to the memory after encapsulation, the reference count of the encapsulated data object is counted as one of the standards for hot and cold data discrimination, which can improve the accuracy of the hot and cold data differentiation processing.

[0044] Further, step S2 connects the three operations of temporary storage of data in the memory, flushing of data from the memory to the hard disk and maintenance of the index, thereby forming a complete and standardized log file. Block writing ensures the efficiency of data writing, and efficient index maintenance ensures the efficiency of data reading.

[0045] Further, the data files in the hard disk will trigger the merging operation when meeting the condition, and the merging operation is divided into the first type of merging and the second type of merging, the merging operation reorganizes the data already written in the hard disk, and continuously adjusts the layout of the data existing in the hard disk, so that the layout is beneficial to the efficient access of hot data.

[0046] Further, the triggering condition of the merging operation is that the number of data files of a layer reaches a preset threshold; and / or, the total size of data files of a layer reaches a preset threshold; and / or, the total number of invalid read of a layer of files reaches a preset threshold; and / or, the space amplification of a layer of files reaches a preset threshold, and the selection of the triggering condition can be very flexible, and for the user, the selection of the triggering condition is the trade-off between read-write performance, read-write amplification and the like.

[0047] Further, step S3 is the management of the data files, and by implementing the pressure of writing data to multiple layers, the overall data files are orderly, and the performance of the read operation of the data files of each layer is also ensured.

[0048] Further, for the first type of merging, the files of level 0 layer and level 1 layer are read, the data in the files is reorganized to generate new files, and the generated new files are placed in level 1 layer, for the second type of merging, the files of level i layer and level i+1 layer are read, i>0, the data in the files is reorganized to generate new files, the file containing hot data is placed in level i layer, and the file containing cold data is placed in level i+1 layer, the difference between the first type of merging and the second type of merging lies in that the data files involved in the two types of merging processes are different, the file formats of level 0 layer and level 1 layer are different, and the data range between the files in level 0 layer may also overlap, so the newly generated files can only be written into level 1 layer, to ensure the correctness of data reading and the efficiency of data access, the data formats of level i layer and level i+1 layer are the same, and the data range between the files in level i layer and level i+1 layer has no overlap, so among the newly generated files, the file containing hot data is placed in level i layer to ensure the access efficiency of hot data, and the file containing cold data is placed in level i+1 layer to vacate space for continuous writing of data.

[0049] It can be understood that the beneficial effects of the above-mentioned second aspect can be referred to the related description in the above-mentioned first aspect, which will not be repeated here.

[0050] In summary, the present application improves the overall access performance of data, effectively alleviates the read amplification problem, and the read performance is very balanced.

[0051] The technical solutions of the present application will be further described in detail below with the aid of the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 A schematic diagram of the method of the present application;

[0053] Figure 2 A schematic diagram of the data file layout in a hard disk under ideal conditions;

[0054] Figure 3 A schematic diagram of the first type of merging;

[0055] Figure 4 A schematic diagram of the second type of merging. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and completely below with the aid of the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0057] In the description of the present application, it should be understood that the terms "comprise" and "include" indicate the presence of described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or sets thereof.

[0058] It should also be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0059] It should be further understood that the term "and / or" used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations, for example, A and / or B can represent three cases of A alone, A and B together, and B alone. In addition, the character " / " in the present application generally represents an "or" relationship between the front and rear associated objects.

[0060] It should be understood that although the terms first, second, third, etc. can be used herein to describe various ranges, etc., these ranges should not be limited to these terms. These terms are only used to distinguish one range from another. For example, a first range could be termed a second range without departing from the scope of the embodiments of the application, and similarly, a second range could be termed a first range.

[0061] Depending on the context, the word "if" as used herein can be interpreted to mean "when" or "while" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]."

[0062] Various structural diagrams according to the embodiments of the present application are shown in the drawings. These diagrams are not drawn to scale, in which certain details are exaggerated for the purpose of clarity and precision, and certain details can be omitted. The shapes of various regions, layers, and their relative sizes and positional relationships shown in the drawings are only exemplary, and in actuality, they can deviate due to manufacturing tolerances or technical limitations, and a person skilled in the art can additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0063] The present application provides a key-value storage method based on LSM tree, which comprises three parts, namely: memory storage structure, based on the distinction between hot and cold data, as cache to support efficient access of data; log file in hard disk, responsible for fault recovery and data persistence two functions, while establishing index to ensure its reading performance; data file in hard disk, composed of multiple files located in multiple levels, dynamically adjusting the layout of hot and cold data by means of merging process. Through the cooperation of the above three parts, the access performance of LSM tree can be improved.

[0064] The present application provides a key-value storage method based on LSM tree, which comprises the following steps:

[0065] S1, memory storage structure, based on the distinction between hot and cold data, as cache to support efficient access of data;

[0066] S101, a certain capacity of space is opened up in the memory to accommodate user data, and the storage structure available for storing data in the embodiment includes one or several of linked list, hash table, tree;

[0067] S102, statistics are made on information of some aspects of data in the storage structure, the information including one or more of access times, access times and semantic relationships between data, in particular, for some data with complex content, the data may be encapsulated and then saved in the memory, at this time, the reference count of the encapsulated data object is also listed as the information to be counted;

[0068] S103, the data is sorted according to the information counted in step S102, the heat of the data is determined according to the sorting result, and the data is divided into hot data or cold data;

[0069] The division of hot and cold data does not have an absolute standard, and the role of sorting the data according to the statistical information is to preferentially remove the data ranked last from the memory when the memory space is full.

[0070] S104, if the capacity of the storage structure in the memory is full, consider eliminating the cold data, and delete one or more data in the memory storage structure;

[0071] S105, after the cold data is eliminated in step S104, the statistical information of the remaining data is all retained or all cleared or periodically cleared.

[0072] The functions of hot and cold data division and cold data elimination are realized, and different hot and cold data division standards and cold data elimination strategies are selected according to different scenes, which flexibly adapts to different workloads. The technical effect of keeping hot data in memory and eliminating cold data when the storage structure capacity is full can be achieved, so that the hot data has good access performance, thereby improving the overall access performance of the LSM tree.

[0073] S2, the log file in the hard disk is responsible for the functions of fault recovery and data persistence, and an index is established to ensure its reading performance;

[0074] S201, a small amount of space in the memory is used for temporarily storing data, and the data residing in the memory includes the key, value field of user data, and the type of operation;

[0075] S202, when the size of the data in the memory reaches a preset threshold, the data is serialized and written into the log file in units of hard disk blocks, and the index of the corresponding data in the memory is updated, the index including the key field and the value field of the data in the log file;

[0076] S203, when the size of the log file reaches a preset threshold, the remaining data in the memory is written into the log file, and the index of the data is written to the end of the log file to form a complete log file;

[0077] S204, the log file is handed over to the data file part in the hard disk for management, and a new file is created as a log file to receive the writing of data stream.

[0078] It can be understood that through the implementation of the above steps, the writing performance and reading performance of the log file are effectively guaranteed, and the functions of fault recovery and data persistence are provided. After the complete log file is created, it is handed over to the data file part in the hard disk for management. In this process, the speed of transition of the log file to the data file can be adjusted to dynamically balance the read-write performance of the LSM tree.

[0079] The data files in the hard disk are multiple and located in multiple different levels. It can be considered that there are n+1 levels from level 0 to level n;

[0080] The files in the level 0 layer are directly converted from the complete log file. The data in the files in the level 0 layer is arranged in time sequence, and the index of the data is located at the end of the file. The data stored in different files in the level 0 layer can have range overlap.

[0081] The files in the level 1 to level n layers are arranged in order in a certain way (usually in lexicographic order). The data stored in different files in the level i (1≤i≤n) layer can not have range overlap.

[0082] The data files in the hard disk will trigger a merge operation when certain conditions are met. The merge operation is a process of reading one or more data files, rearranging the data therein, and writing to a new data file. The merge operation is always performed between adjacent levels. According to the levels of the files participating in the merge operation, the merge operation can be divided into:

[0083] The first type of merge, that is, the files participating in the merge operation are data files in the level 0 and level 1 layers;

[0084] The second type of merge, that is, the files participating in the merge operation are data files in the level i (i>0) and level i+1 layers.

[0085] The trigger conditions of the merge operation include:

[0086] The number of data files in a certain layer reaches a preset threshold;

[0087] And / or,

[0088] The total size of the data files in a certain layer reaches a preset threshold;

[0089] And / or,

[0090] The total number of invalid reads of a certain layer file reaches a preset threshold;

[0091] and / or,

[0092] The space amplification of a certain layer file reaches a preset threshold.

[0093] It can be understood that the closer the layer corresponding to the data file where the data is located to level 0, the better the access performance.

[0094] Please refer to Figure 2 , the layout of the data files in the hard disk should be: the data is sorted from hot to cold according to the heat and then distributed in level 0 to level n layer. In order to make the layout of the data files in the hard disk as close as possible to the layout under the ideal condition, the present application provides an embodiment of the merging operation, which aims to dynamically adjust the layout of the data files in the hard disk to make the hot data obtain better access performance.

[0095] S3, the data files in the hard disk are composed of multiple files located in multiple layers, and the layout of the hot and cold data is dynamically adjusted by means of the merging process.

[0096] The specific steps of the embodiment of the merging operation between the data files in the hard disk are:

[0097] S301, selection of files to be merged, the selection strategies include: polling selection, selection of files with small overlap range, selection of files with low heat, selection of files with more delete / update marks;

[0098] S302, reading the data files of the adjacent two layers, rearranging the data therein to generate new files, and placing the new files to the corresponding layers;

[0099] For the first type of merging, please refer to Figure 3 , when executed, the files of level 0 layer and level 1 layer are read, the data therein is rearranged to generate new files, and the generated new files are placed to level 1 layer;

[0100] For the second type of merging, please refer to Figure 4 , when executed, the files of level i(i>0) layer and level i+1 layer are read, the data therein is rearranged to generate new files, and the file containing hot data is placed to level i layer and the file containing cold data is placed to level i+1 layer;

[0101] S303, retaining the new files generated after merging, and cleaning the invalid old files.

[0102] It can be understood that after the trigger condition of the selected merge operation is selected, the process of the above-mentioned merge operation can be recursive, that is, one merge operation can bring additional merge operations. At the same time, although the merge operation occurs between adjacent levels of data files, a series of merge operations can cause data to move between multiple levels, so the adjustment effect of the hot and cold data layout does not depend on a single merge operation, but depends on the overall situation after multiple merge operations occur.

[0103] In another embodiment of the present application, a key-value storage system based on an LSM tree is provided, which can be used to implement the key-value storage method based on the LSM tree. Specifically, the key-value storage system based on the LSM tree includes a memory module, a log module, and a data module.

[0104] The memory module uses memory space to store hot data and evict cold data.

[0105] The log module manages log files in the hard disk, writes in blocks of integer multiples of hard disk blocks, and reads through indexes in memory to speed up reading.

[0106] The data module arranges data files in the hard disk through merge operations based on a multi-level multi-file structure to implement key-value storage based on the LSM tree.

[0107] In another embodiment of the present application, a terminal device is provided, which includes a processor and a memory. The memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement a corresponding method flow or a corresponding function. The processor in the embodiments of the present application can be used for the operation of the key-value storage method based on the LSM tree, including:

[0108] Hot data is stored in memory space, and cold data is evicted; log files in the hard disk are managed, and when writing, the writing is performed in blocks that are integer multiples of hard disk blocks, and when reading, the reading speed is accelerated through the index in the memory; based on the multi-level multi-file structure, the data files in the hard disk are sorted through the merging operation, and the key-value storage based on the LSM tree is realized.

[0109] In still another embodiment of the present application, the present application further provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in a terminal device, and is used for storing programs and data. It can be understood that the computer readable storage medium herein can include a built-in storage medium in the terminal device, and of course can also include an expansion storage medium supported by the terminal device. The computer readable storage medium provides a storage space, and the storage space stores an operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and the instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory.

[0110] The one or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to realize the corresponding steps of the key-value storage method based on the LSM tree in the above embodiments; the one or more instructions in the computer readable storage medium are loaded and executed by the processor as follows:

[0111] Hot data is stored in memory space, and cold data is evicted; log files in the hard disk are managed, and when writing, the writing is performed in blocks that are integer multiples of hard disk blocks, and when reading, the reading speed is accelerated through the index in the memory; based on the multi-level multi-file structure, the data files in the hard disk are sorted through the merging operation, and the key-value storage based on the LSM tree is realized.

[0112] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0113] The experimental results of the comparison experiment of the corresponding system of the application and the traditional LSM tree show that when the YCSB tool is selected to generate a data set conforming to the zipfian distribution, the corresponding system of the application has a throughput rate increased by 6.3% to 27.3% compared with the traditional LSM tree under different operation quantities, a read amplification reduced by 13.9% to 29.8% under different operation quantities, and a more smooth throughput rate curve for different time periods.

[0114] In summary, the key value storage method and system based on the LSM tree has the following effects:

[0115] By differentiating the hot and cold data, the hot data is in the storage structure of the memory and the upper data file, has better access performance, the cold data is in the lower data file to complete the persistence, while retaining the sequential IO advantage of the LSM tree, the overall access performance of the data is improved.

[0116] When the data is read, the hot data is in the storage structure of the memory and the upper data file, the number of files to be read and the number of hard disk blocks to be read are reduced, and the read amplification problem is effectively alleviated.

[0117] When the data is read, most of the read requests fall on a small amount of hot data, and the hot data has better access performance, so that the response time of most read requests is stable in a specific range, and the read performance in different time periods during the execution process of the read request is very balanced.

[0118] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0119] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0120] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present application can be realized in electronic hardware or in combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0121] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal and method can be implemented in other ways. For example, the apparatus / terminal embodiments described above are merely schematic. The division of the modules or units is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or in other forms.

[0122] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0123] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0124] The integrated module / unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the computer-readable medium can include or exclude contents according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0125] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The flow or multiple flows and / or blocks Figure 1 The apparatus that implements the functions specified in one or more blocks or multiple blocks.

[0126] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The flow or multiple flows and / or blocks Figure 1 The apparatus that implements the functions specified in one or more blocks or multiple blocks.

[0127] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the flowchart Figure 1 one flow or a plurality of flows and / or the functions specified in a block Figure 1 one block or a plurality of blocks.

[0128] The above is only to illustrate the technical idea of the present application, and cannot limit the protection scope of the present application. Any modification made according to the technical idea of the present application on the basis of the technical scheme falls within the protection scope of the claims of the present application.

Claims

1. A key-value storage method based on an LSM tree, characterized in that, Includes the following steps: S1. Use memory space to store hot data and evict cold data; S2. Manage the log files on the hard disk. When writing, write in blocks in multiples of hard disk blocks. When reading, speed up the reading speed by using the index in memory. S3. Based on a multi-level, multi-file structure, data files on the hard disk are organized through merging operations to achieve key-value storage based on an LSM tree; Step S1 is as follows: S101. Allocate space in memory to accommodate user data; S102. Perform statistics on the information of the data in the storage structure. The information of the data in the storage structure includes one or more of the following: access count, access time, and semantic relationships between data. S103. Sort the data according to the information collected in step S102, determine the popularity of the data based on the sorting results, and divide the data into hot data or cold data. S104. When the capacity of the memory storage structure is full, delete one or more data entries located in the memory storage structure. S105. After eliminating cold data in step S104, retain all or delete all statistical information of the remaining data, or delete it periodically. Step S2 is as follows: S201. Temporarily store data in memory. The temporarily stored data includes the key and value fields of user data and the type of operation. S202. When the size of the data in memory reaches a preset threshold, the data is serialized and written to the log file in blocks by disk blocks. At the same time, the index of the corresponding data in memory is updated. The index includes the offset of the key field and value field of the data in the log file. S203. When the size of the log file reaches the preset threshold, first write the remaining data in memory to the log file, and then write the index of the data to the end of the log file to form a complete log file. S204. The log file is handed over to the data file portion of the hard drive for management, and a new file is created as the log file to receive data stream writes. Step S3 is as follows: S301. Selection of files to be merged. Selection strategies include: round-robin selection, selection of files with small overlap, selection of files with low popularity, and selection of files with many deletion / update markers. S302. Read the data files of two adjacent layers, reorganize the data in them to generate a new file, and place it in the corresponding layer; S303. Keep the new file generated after merging and clean up the old files that are no longer valid.

2. The key-value storage method based on LSM tree according to claim 1, characterized in that, In this group S101, the storage structure used to store data includes one or more of the following: linked list, hash table, and tree.

3. The key-value storage method based on LSM tree according to claim 1, characterized in that, In step S102, for the data that has been encapsulated and saved in memory, the reference count of the encapsulated data object is counted.

4. The key-value storage method based on LSM tree according to claim 1, characterized in that, In step S204, the data files on the hard disk trigger a merge operation when certain conditions are met, which is divided into: The first type of merging involves data files from level 0 and level 1. The second type of merging involves data files from level i (i>0) and level i+1.

5. The key-value storage method based on LSM tree according to claim 4, characterized in that, The conditions for triggering the merge operation are: The number of data files in a certain layer reaches a preset threshold; and / or, the total size of data files in a certain layer reaches a preset threshold; and / or, the total number of invalid reads of files in a certain layer reaches a preset threshold; and / or, the space enlargement of files in a certain layer reaches a preset threshold.

6. The key-value storage method based on LSM tree according to claim 1, characterized in that, In step S302, for the first type of merging, the files of level 0 and level 1 are read during execution, the data in them are reorganized and a new file is generated, and the generated new file is placed in level 1. For the second type of merging, during execution, the files of level i and level i+1 are read, i>0. The data in them is reorganized and a new file is generated. The file containing hot data is placed in level i, and the file containing cold data is placed in level i+1.

7. A key-value storage system based on an LSM tree, characterized in that, include: The memory module uses memory space to store hot data and evict cold data, specifically: Allocate space in memory to hold user data; The system performs statistical analysis on the data in the storage structure. This data includes one or more of the following: access count, access time, and semantic relationships between data. Based on the statistical information, the data is sorted, and the data is classified as hot or cold based on the sorting results. When the capacity of the in-memory storage structure is full, one or more data entries are deleted from the in-memory storage structure. After discarding cold data, the statistical information of the remaining data is either fully retained, fully cleared, or periodically cleared. The logging module manages log files on the hard drive. When writing, it writes in blocks of integer multiples of the hard drive size. When reading, it uses in-memory indexes to speed up the reading process. Specifically: Data is temporarily stored in memory, including the key and value fields of user data and the type of operation. When the size of the data in memory reaches a preset threshold, the data is serialized and written to the log file in blocks on the hard disk. At the same time, the index of the corresponding data in memory is updated, including the offset of the key and value fields of the data in the log file. When the size of the log file reaches a preset threshold, the remaining data in memory is written to the log file first, and then the index of the data is written to the end of the log file to form a complete log file. The log file is managed by the data file portion of the hard drive, and a new file is created to receive the data stream writes; The data module, based on a multi-level, multi-file structure, organizes data files on the hard drive through merging operations, implementing key-value storage based on an LSM tree. Specifically: The selection strategies for files to be merged include: round-robin selection, selection of files with small overlap, selection of files with low popularity, and selection of files with many deletion / update tags. Read the data files from the two adjacent layers, reorganize the data to generate a new file, and place it in the corresponding layer; retain the newly generated file after merging, and clean up the invalid old files.

8. The key-value storage system based on LSM tree according to claim 7, characterized in that, The memory module is heavy, and the storage structure used to store data includes one or more of the following: linked list, hash table, tree. For the data that is encapsulated and saved in memory, the reference count of the encapsulated data object is counted.

9. The key-value storage system based on LSM tree according to claim 7, characterized in that, In the log module, data files on the hard drive trigger a merge operation when certain conditions are met, which is divided into: The first type of merging involves data files from level 0 and level 1. The second type of merging involves merging the data files of level i (i>0) and level i+1. The conditions for triggering the merge operation are: The number of data files in a certain layer reaches a preset threshold; and / or, the total size of data files in a certain layer reaches a preset threshold; and / or, the total number of invalid reads of files in a certain layer reaches a preset threshold; and / or, the space enlargement of files in a certain layer reaches a preset threshold.

10. The key-value storage system based on an LSM tree according to claim 7, characterized in that, In the data module, for the first type of merging, the files in level 0 and level 1 are read during execution, the data in them are reorganized and a new file is generated, and the generated new file is placed in level 1. For the second type of merging, during execution, the files of level i and level i+1 are read, i>0. The data in them is reorganized and a new file is generated. The file containing hot data is placed in level i, and the file containing cold data is placed in level i+1.

Images