Log structure tree storage method and system based on access popularity and learning index

Abstract

The present invention provides a log structure tree storage method and system based on access heat and learning index, including: step S1, calculating the access heat of the current SSTable based on positive heat, negative heat, birth timestamp and logical timestamp; step S2, based on the access heat, calculating the average total heat of all SSTables, calculating the corresponding average heat frequency and the relative heat of each SSTable, so as to achieve compression based on access heat; step S3, when the number of KVs key-value pairs stored in the level level exceeds the first preset threshold, calculating the heat sum of the SSTable, and adding each SSTable and its heat sum to the priority queue Q, so as to achieve element dequeue; step S4, performing learning index judgment on the SSTable of the dequeued element, if there is no learning index, determining the preset error and prefix length based on the access heat, and constructing the learning index. The present invention obtains the access heat by calculation, and performs compression optimization and index construction trade-off based on the access heat, so as to improve the overall performance of its storage.

Description

Technical Field

[0001] The present invention relates to a log structure tree storage method, in particular to a log structure tree storage method based on access popularity and learning index, and further to a log structure tree storage system adopting the log structure tree storage method based on access popularity and learning index. Background Art

[0002] With the advent of the era of cloud computing and big data, users' demands for storage are becoming more and more diversified. In this context, key-value (KV) storage, as a popular non-relational (Not Only SQL, NoSQL) database in the world today, is increasingly widely used in the storage field. KV storage stores data in the form of key-value pairs, with each key corresponding to a unique value. This simple data structure gives KV storage the advantages of high performance, strong scalability, ease of use, and high reliability. Since KV storage is not constrained by a fixed table structure, it can store data in any format and is suitable for storing semi-structured and unstructured data. Overall, KV storage has important research value and broad practical value in the era of cloud computing and big data, and has become an important data storage method for large-scale distributed systems and cloud-native applications.

[0003] Although KV storage has many advantages, compared with traditional relational databases, KV storage only provides simple read and write interfaces, but does not have complex query and modification functions. It cannot optimize query and modification operations like Structured Query Language (SQL), which leads to large differences in the time spent on different operations. The main goal of KV storage optimization is to improve the overall read and write performance of the system. In addition, most KV storage tends to be standardized, only providing KV storage functions, but lacking implementation methods for learning user access rules. Therefore, how to make the KV storage system more intelligent, and then perform different internal operations such as compression, index building, etc. in the face of different access frequencies to improve the overall read and write performance of the system is a very valuable research direction.

[0004] In data-intensive systems, log-structured merge trees are usually used as the core index structure for persistent key-value storage. Log-structured merge trees refer to log-structured merge trees, hereinafter referred to as LSM trees or LSM-Trees, also known as log structure trees. Common storage engines such as LevelDB, RocksDB, HBase, and Cassandra all use storage structures based on LSM trees. LSM trees delay write requests and save them in memory, batching write requests in memory on time and writing them to persistent storage media in a sequential manner. This fully utilizes the random read and write speed of memory and the high sequential write bandwidth of persistent storage, giving full play to the advantages of different media.

[0005] In a storage system based on LSM-Tree, read and write performance is most affected by the search process. When a user enters a query key, the system locates the corresponding version of the Value by continuously comparing the Key, usually using a binary search algorithm. Common optimization methods currently include: switching the comparison algorithm, which is a more efficient method than binary search, but this comparison method always has a lower time limit; establishing a range index in memory to narrow the comparison range. This method of exchanging space for time will cause memory consumption to increase sharply as the comparison range decreases. If the compression strategy is improved to reduce the write amplification problem, thereby reducing the consumption of computing resources by internal threads, so that read and write operations obtain more resources, it does not actually directly optimize the read and write operation process, so the overall read and write performance improvement is limited.

[0006] In addition, many high-performance persistent key-value stores are built based on log-structured merge trees, but the inherent index of LSM trees only relies on comparison and does not fully consider the access frequency and access time of data. In addition to the hot information, since the sorted string table SSTable of each layer is not necessarily ordered, the storage system will have problems with low read performance and range query performance. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a log structure tree storage method based on access heat and learning index, aiming to directly optimize the search process in the storage system, integrate a new learning index under read-write intensive workloads, and obtain access heat and relative heat information by calculation, and perform compression optimization and index construction trade-offs based on access heat to improve the read performance and query performance of the storage system, optimize its compression operation, and improve the overall performance of the log structure tree storage method. On this basis, a log structure tree storage system that adopts the log structure tree storage method based on access heat and learning index is further provided.

[0008] To this end, the present invention provides a log structure tree storage method based on access popularity and learning index, comprising the following steps:

[0009] Step S1, calculating the access heat of the current sort string table SSTable based on the positive heat, negative heat, birth timestamp and logical timestamp;

[0010] Step S2, based on the access heat of the current sort string table SSTable, calculate the average total heat of all sort string tables SSTables, and calculate the corresponding average heat frequency and the relative heat of each sort string table SSTable to achieve compression based on access heat;

[0011] Step S3: When the number of KVs key-value pairs stored in the level exceeds the first preset threshold, the heat and Hot value of each sorted string table SSTable in the level are calculated. sum , and each sorted string table SSTable and its corresponding heat and Hot sum Add to Heat and Hot sum Sorted priority queue Q, heat and Hot sum The sorting is in descending order, and the heat and Hot sum The sort string table SSTable at the head of the sorting team is dequeued;

[0012] Step S4, for each sorted string table SSTable of the dequeued element, determine whether there is a learned index Learned Index. If not, determine the preset error error and prefix length length used to construct the learned index Learned Index based on the access popularity, and construct the learned index Learned Index; if so, exit and wait for the next round of judgment.

[0013] A further improvement of the present invention is that in step S1, by formula Calculate the access heat of the current sort string table SSTable get , among which, Hot p Indicates positive heat, Hot n represents the negative heat, α and β are the preset positive heat weight and negative heat weight respectively, T now Represents the logical timestamp, T Start Represents the birth timestamp.

[0014] A further improvement of the present invention is that, in step S1, for the newly generated sort string table SSTable, the formula Calculate the inherited heat of the compressed set corresponding to the newly generated sorted string table SSTable inherent , thereby inheriting the heat Hot inherent As the access popularity of the newly generated sort string table SSTable, size represents the number of KVs key-value pairs of the newly generated sort string table SSTable, size all Indicates the number of KVs key-value pairs in the compressed collection, Hot all Represents the sum of the access heats of all sorted string tables SSTables in the compressed set.

[0015] A further improvement of the present invention is that in step S2, by formula Calculate the average total heat of all sorted string tables SSTables Hot avg , among which, Hot i =Hot get +Hot inherent , Hot i Represents the sum of the Get operation heat and the inheritance heat of the i-th sorted string table SSTable in level level, and n represents the number of all sorted string table SSTables.

[0016] A further improvement of the present invention is that in step S2, by formula Calculate the average heat frequency Tp i , and by the formula Average heat frequency Tp i Perform heat normalization to obtain the normalized heat frequency Tp avg .

[0017] A further improvement of the present invention is that in step S2, by formula Calculate the relative heat of each sort string table SSTable level-i , select relative heat Hot level-i The smallest sorted string table SSTable is used as the string table to be compressed.

[0018] A further improvement of the present invention is that in step S3, the formula Hot sum =γ*Hot get +δ*Hot inherent Calculate the heat and hotness of each sorted string table SSTable in level level sum , where γ and δ are the preset access heat weight and inherited heat weight respectively.

[0019] A further improvement of the present invention is that in the step S4, it is determined whether the access heat of the sorted string table SSTable is greater than a second preset threshold value. If so, the value of the preset error error of the learned index Learned Index is reduced, and its prefix length length is increased; if not, the value of the preset error error of the learned index Learned Index is increased, and its prefix length length is reduced; the process of constructing the learned index Learned Index is: first, assume that the elements of the selected training data set D are tuple type D i =(K i , P i ), D i represents the i-th tuple in the training data set D, K i and P i Respectively represent the i-th key key and the corresponding logical address in the structure data SSTable to be trained; then through the formula S(K i )=P i ±error trains the interpolation point model S, where error represents the preset error; finally, the prefix search array and interpolation points are combined for query.

[0020] A further improvement of the present invention is that the query process combining the prefix search array and the interpolation point is as follows: access the pointer corresponding to the logical address and the prefix in the prefix search array, and if the corresponding interpolation point P1 is found, then start from the interpolation point P1 and gradually traverse the interpolation point sequence backward and find the last one that is not greater than the Get operation index Key get interpolation point P2; otherwise, end the query; after finding the interpolation point P2, use the interpolation point P2 and the next interpolation point P3 next to the interpolation point P2 as two base points, perform interpolation operations on these two base points, obtain the predicted logical position Position, and access the index Key corresponding to the predicted logical position Position p , if Key p =Key get , the query ends; if Key p ≠Key get , then use binary search to query in the range of (Position-erroe,Position+error).

[0021] The present invention also provides a log structure tree storage system based on access heat and learning index, which adopts the log structure tree storage method based on access heat and learning index as described above, and includes:

[0022] The access heat calculation module calculates the access heat of the current sort string table SSTable based on positive heat, negative heat, birth timestamp and logical timestamp;

[0023] The compression optimization module based on access heat calculates the average total heat of all sorted string table SSTables based on the access heat of the current sorted string table SSTable, and calculates the corresponding average heat frequency and the relative heat of each sorted string table SSTable to achieve compression based on access heat;

[0024] The element dequeue judgment module based on access heat calculates the heat and hotness of each sorted string table SSTable in the level when the number of KVs key-value pairs stored in the level exceeds the first preset threshold. sum , and each sorted string table SSTable and its corresponding heat and Hot sum Add to Heat and Hot sum Sorted priority queue Q, heat and Hot sum The sorting is in descending order, and the heat and Hot sum The sort string table SSTable at the head of the sorting team is dequeued;

[0025] The index construction weighing module based on access popularity determines whether there is a learned index Learned Index for the sorted string table SSTable of each dequeued element. If not, the preset error error and prefix length length used to construct the learned index Learned Index are determined based on the access popularity, and the learned index Learned Index is constructed; if so, exit and wait for the next round of judgment.

[0026] Compared with the prior art, the beneficial effects of the present invention are: it aims to directly optimize the search process in the storage system, integrate a new learning index under read-write intensive workloads, first calculate the access heat of the current sort string table SSTable based on positive heat, negative heat, birth timestamp and logical timestamp; then, based on the access heat of the current sort string table SSTable, calculate the average total heat of all sort string table SSTables, and calculate the corresponding average heat frequency and the relative heat of each sort string table SSTable to achieve compression based on access heat, and achieve compression optimization based on access heat, which can effectively avoid the phenomenon of large fluctuations in Get operation performance; then, after the number of KVs key-value pairs stored in a certain level exceeds the first preset threshold, calculate the heat and Hot of each sort string table SSTable in the level based on the access heat. sum , get the corresponding priority queue Q, and add the heat and Hot sumThe sorted string table SSTable at the head of the sorted queue is dequeued; finally, the sorted string table SSTable of each dequeued element is weighed based on the access popularity of the index construction. Therefore, the present invention can effectively improve the reading performance and query performance of the storage system, and effectively avoid the waste of resources, so as to comprehensively improve the overall performance of the log structure tree storage method and the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the overall architecture of an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of the merging process of the KV database engine;

[0029] Figure 3 It is a schematic diagram of the working process of an embodiment of the present invention;

[0030] Figure 4 It is a schematic diagram of an index according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In the description of the present invention, if "several" is involved, it means more than one; if "a certain" is involved, it means at least one of them; if "multiple" is involved, it means more than two; if "greater than", "less than", "exceed", they should be understood as not including the number; if "above", "below", "within" are involved, they should be understood as including the number. If "first", "second", etc. are involved, they should be understood as being used only to distinguish the names of the same or similar technical features, and cannot be understood as implying / indicating the relative importance of the technical features, the number of technical features, or the order of the technical features.

[0032] Before describing the specific implementation methods of the present application, this embodiment first describes the relevant abbreviations, term definitions and technical solutions.

[0033] Definitions of abbreviations and key terms are as follows:

[0034] LSM-Tree refers to Log-Structured Merge-tree, that is, the log-structured merge tree is a data structure, referred to as the log structure tree in this application, and its performance characteristics make it suitable for providing index access to files with high insertion volume, such as transaction log data.

[0035] LevelDB is a persistent KV database engine developed and open sourced by Google engineers Jeff Dean and Sanjay Ghemawat. MemTable refers to a memory table, a data structure that stores data in memory before flushing it to disk.

[0036] SSD refers to Solid-state disk, abbreviated as SSD. It is a computer storage device made of integrated circuits. Although the price and storage capacity are slightly different from mechanical hard disks, the reading speed of solid-state drives can be 200 times faster than that of mechanical hard disks.

[0037] SSTable refers to Sorted String Table, which is the abbreviation of sorted string table and comes from the Google Bigtable paper. It is used to store data files inside Bigtable and is an efficient key-value file storage format.

[0038] The learned index is a data search structure first proposed by Google. It generates a search model by learning the distribution of the data to be searched, thereby supporting fast search, insertion, and deletion of data.

[0039] B+ tree is a tree data structure, usually used in database and operating system file systems. The characteristic of B+ tree is that it can keep data stable and orderly, and its insertion and modification have a relatively stable logarithmic time complexity.

[0040] Access popularity, each SSTable has access popularity, that is, users' different access preferences, which are specifically expressed as the frequency of access.

[0041] A technical solution related to the present invention is the LSM tree structure.

[0042] The LSM tree structure is proposed to optimize the write performance of key-value storage systems. In this data storage structure, new data write operations will only be written to the disk in a sequential manner. Data write operations include inserting new records, updating records, and deleting records. That is to say, under write-intensive workloads, operations related to data writes will be completed in a way that optimizes disk performance, fully optimizing the system's write efficiency. In the LSM tree storage model, when the system receives a write operation, it first saves the records corresponding to this write operation in memory. When the size of all records cached in memory is greater than a certain limit, these records will be integrated into a table file and the table file will be written to the disk in a sequential manner. The data structure composed of records cached in memory is called MemTable. In LevelDB, MemTable is organized into a skip-list data structure to facilitate fast search and insertion of records. When the size of MemTable reaches a certain limit, all records in it will be written to a file of a specific structure in sequence according to the sorted size, and then the file will be written to the disk sequentially. In the LSM tree structure, all data storage files have the same layout, and such files are called SSTable. In SSTable, records are arranged in order from small to large. This layout is conducive to searching for data in the table file, so that the search for specific keys can be performed in a binary search manner.

[0043] When the size of the records in the MemTable gradually grows to the limit that needs to be written to the disk, all the records will first be constructed into an SSTable, and then the table file will be written to the disk. In the partitioning merge policy pioneered by LevelDB, the disk components of each level (Level) are divided into multiple fixed-size SSTables by range. Each SSTable is marked with its key range. In order to merge a Level L SSTable into a Level L+1 SSTable, all its overlapping Level L+1 SSTables will be selected and merged with it to generate a new SSTable still at Level L+1. For example, Figure 2As shown, the SSTables marked 0-30 at Level 1 are merged with the SSTables marked 0-15 and 16-32 at Level 2. This merge operation generates new SSTables marked 0-10, 11-19, and 20-32 at Level 2, and then the old SSTables will be garbage collected. Under this horizontal partition merge strategy, the merge from the upper layer to the lower layer may affect multiple SSTables at the next layer, causing these SSTables to need to be read and written once, which will cause serious write amplification. SSTables refers to multiple SSTables, that is, the plural number of SSTable.

[0044] The basic data operations and their main execution flow in the LSM tree are as follows: Get For a record acquisition operation, the LSM tree structure will return the value of the latest version of the record. Since the LSM tree is a hierarchical structure, and the table files are arranged in order in each layer, the search for a record always starts from the lowest layer, and a binary search between table files and a binary search within the table are performed in each layer. If the record does not exist, the query process starts at the next layer until the record is obtained. In the physical data structure of the LSM tree, the key of each record has a sequential number representing its version number, which is conducive to identifying and operating between multiple versions of a key.

[0045] Iterator, that is, iterator. In the LSM tree structure, iterative queries and range queries on records are completed in a multi-level iterative merge manner. During the query process, an iterator is first built at each level. These iterators will read the table files and data records in the hierarchy in turn. After that, the record keys located by each iterator will be merged and sorted. The obtained record sequence is the result of the global iterator.

[0046] Put, when a new record update is written to the system, the record will first be saved in MemTable, then written to SSTable and enter layer 0. Record updates include data insertion, update and deletion operations. In data deletion operations, the key of the corresponding record is the key to be deleted, and the value is a special deletion flag. When reading a record, when this flag is obtained, it can be known that the current version of the record key and all previous versions of the value have been deleted.

[0047] This related technical solution has the following disadvantages: For storage systems based on LSM-Tree, the search process has the greatest impact on read and write performance. After the user enters the queried Key, the system uses the input Key to continuously compare, generally using a binary search method, and returns the Value after finding the corresponding version of the Key. At present, common optimization methods include: switching the comparison algorithm, using this more efficient comparison method than binary search, but this algorithm still uses the comparison method, and there will always be a problem that the lower limit of time cannot be broken. If a range index is established in the memory to narrow the comparison range, but this method of exchanging space for time, as the comparison range is narrowed, the memory consumption increases sharply; if the compression strategy is improved to reduce the write amplification problem, thereby reducing the consumption of computing resources by internal threads, so that read and write operations obtain relatively more resources, but in essence, this method does not directly start from the process of read and write operations, and has limited improvement on the overall read and write performance of the system.

[0048] Another technical solution related to the present invention is a stage-by-stage heat statistics method.

[0049] Yu JP et al. proposed a simple staged heat statistics method, where the number of accesses per w is used as a statistical range, and the current total number of accesses is c. If SStable N is accessed at a certain moment, then access(i) = 1, otherwise access(i) = 0. At this time, the access heat of the SStable is The whole formula shows that the current heat of SStable N is the proportion of the number of accesses within the current window range. Yu JP uses logical timestamps to calculate heat, and sets the global logical timestamp to the current total number of Gets C. Each statistical target has a death timestamp D and a life cycle L. As long as the target serves the Get operation, D=C is executed. At any time, as long as D+L>=C is satisfied, it indicates that the current target is hot data, otherwise it is cold data.

[0050] This technical solution uses the external machine learning prediction model Leaper to predict the popularity of KVs key-value pairs. The popularity prediction can be divided into three components: learning component (Learner), prediction component (Prefetcher) and collection component (Collector). Among them, Learned is an offline learning component deployed on other systems to learn the access popularity of data. Collector is used to selectively collect data that needs to be judged for popularity. Finally, Learner and Collector send the learning model and collection information to Prefetcher to obtain the prediction results. However, the overall architecture of Leaper in this technical solution is too large, and even requires the use of cloud storage servers, which is not conducive to maintaining data security and reliability.

[0051] In addition, there are the following shortcomings: The heat information statistics for KVs key-value pairs are not comprehensive, for example, SStables misses are ignored. Although some SStables have frequent active queries, there are likely to be many passive queries as well, so the comprehensive heat needs to be analyzed in combination with active and passive queries, and the existing staged heat statistics methods obviously do not do this. At the same time, the generation time and source of the SStable will also affect the heat, and these factors cannot be ignored.

[0052] To this end, the present invention aims to provide a log structure tree storage method and system based on access popularity and learned index, that is, to provide a method for integrating machine learning technology into LevelDB to construct a learned index Learned Index. The log structure tree storage method aims to directly optimize the search process in the storage system, and replace the traditional comparison-based database index structure with the learned index Learned Index to establish a corresponding learned index Learned Index for the smallest unit in LevelDB, SSTable. When searching for the key entered by the user, the key is also called an index, and a trained function, namely Learned Index, is used to predict the location of the key. If successful, this method can complete the read operation in constant time and potentially reduce space overhead; even if it is unsuccessful, compared with the traditional comparison method, it can significantly reduce unnecessary comparison operations, thereby greatly improving the overall performance of the system.

[0053] In addition, since building the Learned Index requires a certain amount of time, not all SSTables are suitable for building this learning index. At the same time, there are significant differences between the various SSTables at different access frequencies, so it is necessary to more flexibly and intelligently determine which SSTables should build the Learning Index. To meet this demand, the present invention proposes a delayed compression strategy based on access heat. The delayed compression strategy calculates the access heat, such as counting user access log data, including read, write and delete operations. When performing compression, the access heat of each SSTable is evaluated based on statistical data, and the SSTables with low heat (i.e., less access) are compressed to the lower layer, while the high-heat SStable is retained at the upper layer and the Learning Index is built for it. Therefore, the technical solution provided by the present invention can implement diversified compression processes and Learning Index construction according to different user access frequencies, realizing the true intelligence of the storage system.

[0054] The present invention provides a log structure tree storage method and system based on access heat and learning index, that is, a key value access heat statistics and index construction mechanism of LSM-Tree. The main technical contents include the following three aspects:

[0055] First, the construction of SSTable learning index - single traversal learning index based on prefix search. The traditional technical solution is based on the comparison index mechanism, such as B-tree, and its algorithm complexity can never break through O(logN). When the LevelDB search deepens with the level hierarchy, the retrieval time will increase sharply, so the present invention aims to establish a new and efficient index mechanism, which is the key to solving the technical problem that the retrieval time will increase sharply with the deepening of the level hierarchy. The present invention integrates the idea of ​​machine learning into the index construction of LevelDB, and designs a single traversal learning index based on prefix search. The learning index can realize fast query of Key within a specific preset error range, and the computational complexity reaches O(1), which greatly improves the performance of Get operation and improves the query performance of the system. The detailed implementation process will be described in detail through step S4 later. The preset error can be set and adjusted according to actual conditions and needs.

[0056] Second, SSTable access heat statistics strategy. The existing LevelDB design process ignores the heat information of the stored KVs, which makes the system's internal compression operations easily have a negative impact on the user's query operations, such as causing large fluctuations in the Get operation performance of different Keys; or the need to use external components to consume a large amount of resources. The present invention provides a low-consumption and high-efficiency embedded SSTable heat statistics strategy, which implements an embedded lightweight access heat statistics strategy that does not require external components; the detailed implementation process will be described in detail later through step S1.

[0057] Third, the trade-off between compression optimization and index construction based on access popularity. The compression operation of LevelDB follows a cyclic order, and each SSTable has an equal opportunity to enter the compression operation. This existing design ignores the key value access popularity of different SSTables. If the frequently accessed SSTable is pressed into a lower level, the performance of the Get operation will be greatly reduced. In addition, the index construction of SSTable consumes system resources, and not all SSTables are worth building an index, so a reasonable and efficient index construction trade-off mechanism is required for judgment. In contrast, the present invention provides a compression optimization and index construction trade-off strategy based on access popularity, which keeps SSTables with high access popularity at a high level as much as possible; on this basis, not all SStables need to build a learning index, so an effective compression optimization and index construction trade-off mechanism is also provided to solve the above problems. The detailed implementation process will be described in detail later through steps S2 to S4.

[0058] The preferred embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings.

[0059] like Figure 1 , Figure 3 and Figure 4 As shown, this embodiment provides a log structure tree storage method based on access popularity and learning index, including the following steps:

[0060] Step S1, calculating the access heat of the current sort string table SSTable based on the positive heat, negative heat, birth timestamp and logical timestamp;

[0061] Step S2, based on the access heat of the current sort string table SSTable, calculate the average total heat of all sort string tables SSTables, and calculate the corresponding average heat frequency and the relative heat of each sort string table SSTable to achieve compression based on access heat;

[0062] Step S3: When the number of KVs key-value pairs stored in the level exceeds the first preset threshold, the heat and Hot value of each sorted string table SSTable in the level are calculated. sum , and each sorted string table SSTable and its corresponding heat and Hot sum Add to Heat and Hot sum Sorted priority queue Q, heat and Hot sum The sorting is in descending order, and the heat and Hot sumThe sort string table SSTable at the head of the sorting queue is dequeued; the first preset threshold refers to the trigger threshold when the number of pre-set KVs key-value pairs reaches the compression condition, which can be set and adjusted according to actual conditions and needs;

[0063] Step S4, for each sorted string table SSTable of the dequeued element, determine whether there is a learned index Learned Index. If not, determine the preset error error and prefix length length used to construct the learned index Learned Index based on the access popularity, and construct the learned index Learned Index; if so, it indicates that the learned index Learned Index already exists, then exit and wait for the next round of judgment.

[0064] Step S1 described in this embodiment is used to implement the SSTable access heat statistics strategy.

[0065] Compared with the prior art, the heat statistics strategy for access heat in this embodiment takes more factors into consideration, including but not limited to whether the access category is active or passive, the time from the most recent active access to the SSTable, etc.

[0066] Next, the calculation process of access popularity in this embodiment will be described in detail. Since the user's access has a large time error, the time difference between two accesses is completely random. In order to remove the impact of the time difference in reality, this embodiment uses the logical timestamp T now Indicates the current time of the system. When the SSTable is created, it will be given a birth timestamp T Start , the last active access timestamp T last_p , the last negative access time T last_n 、Positive heat Hot p and negative heat n , execute T Start =T now 、T last =0, T last_n =0, Hot p =0 and Hot n =0 are the birth timestamps T Start , the last active access timestamp T last , the last passive access timestamp T last_n 、Positive heat Hot p and negative heat n Assign a value.

[0067] When an SSTable is accessed, there are two situations: either the query succeeds or fails, which are called active access and passive access respectively. Obviously, the more active accesses an SSTable has, the hotter it is; the more passive accesses it has, the more it should be compressed to the lowest level, and the lower its heat is. When setting each active access to an SStable, the formula Hotp+=Scorep is used to increase the active heat by a fixed score, and the timestamp of the last active access T is updated. last_p =T now Similarly, for each negative access, the formula Hot n +=Score n Increase the negative heat by a fixed fraction and update the last negative access timestamp T last_n =T now .

[0068] At the same time, in the process of calculating the heat, the SSTable that was recently accessed is considered to be hotter. This embodiment calculates the timestamp with the current system timestamp (i.e., the logical timestamp T now ), and use this time difference to cool the current heat to a certain extent. The cooling formulas are and Among them, k is a pre-set cooling coefficient, which can be set and adjusted according to actual conditions and needs in practical applications.

[0069] After cooling, in step S1 of this embodiment, the formula Calculate the access heat of the current sort string table SSTable get , among which, Hot p Indicates positive heat, Hot n represents negative heat; α and β are the preset positive heat weight and negative heat weight respectively, which can be set and adjusted according to actual conditions and needs; T now Represents the logical timestamp, T Start Indicates birth timestamp; T now -T start Indicates the current survival time of the SSTable.

[0070] On this basis, considering that the newly generated SSTable is merged and sorted by different SSTables (hereinafter referred to as compressed sets), the initial heat of the newly generated SSTable should be related to the compressed set that generated it. This embodiment proposes a heat inheritance strategy, that is, the newly generated SSTable will inherit part of the heat of the corresponding compressed set.

[0071] Therefore, preferably, in step S1 of this embodiment, for the newly generated sorted string table SSTable, the formula Calculate the inherited heat of the compressed set corresponding to the newly generated sorted string table SSTable inherent , thereby inheriting the heat Hot inherent As the access popularity of the newly generated sort string table SSTable, size represents the number of KVs key-value pairs of the newly generated sort string table SSTable, size all Indicates the number of KVs key-value pairs in the compressed collection, Hot all Represents the sum of the access heat of all sorted string tables SSTables in the compressed set, that is, the total access heat.

[0072] From the above, it can be seen that the SSTable access heat statistics strategy of this embodiment takes into account the positive and negative categories of queries, and the classified heat statistics distinguish the SSTable access heat more thoroughly. This embodiment adopts a logical timestamp, thereby reducing the unpredictable real time difference caused by the randomness of user access. At the same time, the heat cooling based on the LRU concept and Newton cooling method helps to screen out the SSTables that have been accessed more frequently recently. In addition, the heat inheritance strategy is used to avoid letting each newly generated SSTable start from 0 heat, making the heat statistics strategy more coherent, more discriminative, and more reasonable. The access heat statistics algorithm provided in this embodiment uses a suitable data structure to eliminate complex loop traversal processes, and only involves simple addition, subtraction, multiplication, and division calculations, thereby achieving the purpose of high efficiency with low complexity.

[0073] In this implementation, steps S2 to S4 are used to achieve compression optimization and index construction trade-offs based on access popularity.

[0074] The purpose of LevelDB compression is to select some SSTables to be compressed into a lower layer when the number of KVs key-value pairs stored in a certain level exceeds the threshold, so as not to affect the overall performance of the system. The original compression strategy of LevelDB did not take into account the access popularity of SSTable, causing some SSTables with high popularity and frequent access to be pressed into a lower layer, resulting in large fluctuations in Get operation performance. The main idea of ​​the compression optimization based on access popularity in this embodiment is to press SSTables with relatively low popularity into a lower layer, and keep SSTables with relatively high popularity at a high level, so as to achieve continuous, stable and efficient Get operation performance. The compression strategy further involves the comparison of the popularity of SSTables at the same level. This embodiment combines the survival time T of the SSTable now -T start、Hot get and Hot inherent The three factors use the idea of ​​Bayesian mean to calculate the relative heat of all SSTables in the same level.

[0075] Specifically, in step S2 of this embodiment, by formula Calculate the average total heat of all sorted string tables SSTables Hot avg , the average total heat Hot avg Also called mean heat. i =Hot get +Hot inherent , Hot i Represents the sum of the Get operation heat and the inheritance heat of the i-th sorted string table SSTable in level level, and n represents the number of all sorted string table SSTables.

[0076] Preferably, this embodiment also normalizes the heat data. In step S2 of this embodiment, the formula Calculate the average heat frequency Tp i , and by the formula Perform heat normalization on the average heat frequency Tpi to obtain the normalized heat frequency Tp avg , thereby preventing the problem of large calculation differences caused by different division calculation bases.

[0077] Finally, in step S2 of this embodiment, by formula Calculate the relative heat of each sort string table SSTable level-i , select relative heat Hot level-i The smallest sorted string table SSTable is used as the string table to be compressed, that is, the SStable to be compressed, to achieve compression based on access heat. Therefore, in step S2 of this embodiment, after obtaining the access heat Hot get Based on the corresponding relative heat Hot level-i To perform compression optimization, thereby effectively avoiding the phenomenon of large fluctuations in Get operation performance.

[0078] Since building a learning index takes a certain amount of time. If each SSTable is indexed, it may happen that the index is built but the corresponding SSTable is compressed, wasting computing resources. Therefore, the trade-off for index construction in this embodiment is also based on access popularity, so as to train SSTables with a long survival time and high popularity as much as possible, effectively avoiding unnecessary waste of computing resources. To this end, this embodiment proposes a compression optimization and index construction trade-off strategy based on access popularity.

[0079] When the compression condition of a certain level is triggered, that is, the number of KVs key-value pairs stored in a certain level exceeds the first preset threshold, in step S3 of this embodiment, the formula Hot sum =γ*Hot get +δ*Hot inherent Calculate the heat and hotness of each sorted string table SSTable at this level sum , where γ and δ are the preset access heat weight and inherited heat weight respectively. The access heat weight and inherited heat weight can also be set and adjusted according to actual conditions and needs.

[0080] The SStable together with its corresponding Hot sum Join Hot sum Sorted priority queue Q, where Hot sum The larger the SStable, the more it is at the head of the queue. Then, according to the preset strategy, the elements in the priority queue Q are dequeued to determine whether the dequeued elements have been built into the LearnedIndex. If not, the error and prefix length of the LearnedIndex are determined using information such as access popularity, and then the index is built.

[0081] like Figure 1 As shown in FIG. 1 , the overall architecture diagram of this embodiment mainly includes three components: an access heat statistics component, a learning index construction component, and a compression optimization and index trade-off component. The access heat statistics component corresponds to step S1, which is used to timely count the access data of different SSTables. Figure 1 In the example, the dashed line with an arrow pointing to the access heat statistics component is used to calculate the heat for each SStable. When compression is triggered, corresponding to steps S2 and S3, Figure 1 The dotted line with an arrow in the middle indicates that the compression metadata is passed to the compression optimization and index trade-off component by the compression thread, and the corresponding heat is obtained from the heat component based on these metadata; after obtaining the corresponding heat, Figure 1As shown in the figure, the access heat statistics component points to the learning index construction component and the compression optimization and index trade-off component respectively, and then selects the SStable that is most suitable for compression to achieve the purpose of optimizing the compression operation; at the same time, it is also necessary to determine whether there is an SSTable that needs to build a learned index, corresponding to step S4. If necessary, the SStable metadata represented by the short dashed line with an arrow issued by the compression optimization and index trade-off component is sent to the learning index construction component. The learning index construction module obtains the heat information of the SStable from the heat component based on the above metadata for setting the parameters of the learning index, and then starts an additional thread to build the learning index. Figure 1 In the MemTable, MemTable refers to a memory data structure used in databases and storage systems, which is mainly used to temporarily store recently written data for fast access and efficient writing; Immu Table refers to Immutable Table, that is, a data table that only allows insertion operations.

[0082] Therefore, this embodiment determines whether it is necessary to build a learned index based on the trade-off between compression optimization and index construction based on access popularity. When the sorted string table SSTable of the dequeued element has a learned index Learned Index, exit and wait for the storage system operation. When the sorted string table SSTable of the dequeued element does not have a learned index Learned Index, step S4 described in this embodiment is also used to implement a single traversal learned index based on prefix search.

[0083] The single-pass learning index based on prefix search aims to map the Key to be queried to the logical address in the SSTable where the Key is located. For example, there is a Key with a value of 200 in SSTable N, which is located at the 20th position (starting from the 0th position) in order, and the logical address is 200. In general, the learned index Learned Index of this embodiment can be divided into two levels: the lower level is a fixed sequence of interpolation points within the preset error range, where error is a preset integer, and the higher level is a prefix search array. The interpolation points are selected according to the pre-strategy Keys, which belong to the part of the Keys set in the currently trained SSTable; the index number of the prefix search array is the prefix of the prefix search array, and the Keys can be calculated using the preset conditions. The preset conditions can be set according to the actual situation and requirements, and do not belong to the scheme of this application and will not be expanded; the data stored in the prefix search array is the pointer to the current interpolation point. The interpolation point sequence is used to meet the preset error range, that is, to query any Key between two interpolation points, the predicted position of the interpolation sequence will not exceed the error from the correct position. The prefix search array is used to quickly locate the range of interpolation points that contain the queried Key.

[0084] In the query, first, according to the prefix of the Key value to be queried, the prefix search array is used to find the correct interpolation point containing the Key value, and then the interpolation algorithm is executed according to the selected interpolation point to obtain the predicted position. Check whether the predicted position is the Key. If so, the query is successful; otherwise, a binary search is performed within the error range. At this time, since the error range is fixed, the search time complexity is O(1).

[0085] Unlike other learning indexes, the learning index proposed in this embodiment only needs to be built in one traversal. The learning index of existing solutions often does not consider the impact of the data volume and access popularity of SSTables in LevelDB on parameter settings. Usually, the size of the error is proportional to the query time and inversely proportional to the space occupied by the index, and the size of the prefix length is inversely proportional to the query time and proportional to the space occupied by the index. This embodiment proposes an adaptive parameter adjustment strategy based on the data volume and access popularity of SSTable for the construction of the learning index. According to this strategy, different SSTables will adaptively set the error and prefix length according to their own data volume and access popularity when building the index. The main technical solution refers to that if the heat of the SSTable is greater than a certain threshold, a smaller error and a larger prefix length are adopted; otherwise, a larger error and a shorter prefix length are adopted.

[0086] Specifically, in step S4 of this embodiment, it is determined whether the access heat of the sorted string table SSTable is greater than the second preset threshold value. If so, the value of the preset error error of the learned index Learned Index is reduced, and its prefix length length is increased; if not, the value of the preset error error of the learned index LearnedIndex is increased, and its prefix length length is reduced. The second preset threshold value refers to a pre-set access heat threshold value, which is used to adaptively set the preset error error and prefix length length according to the size of its own data and the access heat, and can be customized and adjusted according to actual conditions and needs.

[0087] For the construction of the learning index, the learning index proposed in this embodiment adopts a "bottom-up" construction mode. First, traverse the sorted Keys and their corresponding logical addresses in the SStable, select interpolation points from them, and then determine whether the selected interpolation points are added to the prefix search array. The following is a detailed introduction:

[0088] The process of constructing the learned index in this embodiment is as follows: the structure of the learned index is used as a model that maps from a key Key to a logical address. First, the elements of the selected training data set D are tuple type D i =(K i , P i ), D i represents the i-th tuple in the training data set D, K i and P i Respectively represent the i-th key key and the corresponding logical address in the structure data SSTable to be trained; then, through the formula S(K i )=P i ±error trains the interpolation point model S within the error range, where error represents the preset error, so that the interpolation point model S can predict the correct position of the Key in the SSTable within the error range; finally, the prefix search array and interpolation point are combined for query.

[0089] The interpolation point model S is actually a sequence of interpolation points, and the set of all interpolation points is a subset of D. How to select interpolation points to satisfy the correct prediction within the error range is the key to establishing model S. In this embodiment, the spline interpolation algorithm can be selected for interpolation point selection. The algorithm satisfies that the linear interpolation of the two interpolation points closest to any query key value does not exceed the error. The prefix search array plays a huge role in finding the correct interpolation point. The array mainly stores pointers to the interpolation points, and the number is 2 r, where r is the prefix length pre-set according to information such as SSTable access heat.

[0090] When selecting an interpolation point, it is necessary to determine whether the current interpolation point can become an index point in the prefix search array. Compare the prefix of the current interpolation point with the previous interpolation point. If the two are different, the current interpolation point is selected as a new index point in the prefix search array, that is, the pointer in the prefix search array with the same logical address as the prefix of the current interpolation point points to the current interpolation point.

[0091] The query process of this embodiment in combination with the prefix search array and the interpolation point is as follows: access the pointer corresponding to the logical address and the prefix in the prefix search array, and if the corresponding interpolation point P1 is found, then traverse the interpolation point sequence backward step by step from the interpolation point P1 and find the last one that is not greater than the Get operation index Key get interpolation point P2; otherwise, end the query; after finding the interpolation point P2, use the interpolation point P2 and the next interpolation point P3 next to the interpolation point P2 as two base points, perform interpolation operations on these two base points, obtain the predicted logical position Position, and access the index Key corresponding to the predicted logical position Position p , if Key p =Key get , the query ends; if Key p ≠Key get , then use binary search to query in the range of (Position-erroe,Position+error).

[0092] More specifically, the search process combining the prefix search array and the interpolation point is as follows Figure 4 As shown. Need to query Key get is 47180. Assuming that the preset prefix length is 3, the corresponding prefix is ​​5 according to the preset. Access the pointer with the logical address 5 in the prefix search array, find the first interpolation point P1 with the prefix 5, and traverse the interpolation point sequence backward from P1 to find the last one that is not greater than Key. get In this process, if the prefix of the interpolation point queried exceeds 5, it means that there is no Key in the SSTable. get , the query ends immediately. If P2 is successfully found, P2 and the next interpolation point P3 are used as two base points for interpolation operation to obtain the predicted logical position Position. Access the Key of Position p , if it happens to be Key get, the query ends. Otherwise, use binary search to query in the range of (Position-erroe,Position+error). The part with serial number ① in the figure represents the interpolation point found from the prefix search array, and the dotted arrow with serial number ② represents the range of selecting the correct interpolation point. If it exceeds this range, it means that the SSTable does not contain the queried Key. get The circle within the dotted arrow range represents the predicted position after interpolation, and the dotted box within the dotted arrow range represents the binary search within the 2*error range. Finally, the query key pointed by the solid arrow indicates that after the query is successful, the key get The specific location.

[0093] This embodiment also provides a log structure tree storage system based on access popularity and learning index, which adopts the log structure tree storage method based on access popularity and learning index as described above, and includes:

[0094] The access heat calculation module calculates the access heat of the current sort string table SSTable based on positive heat, negative heat, birth timestamp and logical timestamp;

[0095] The compression optimization module based on access heat calculates the average total heat of all sorted string table SSTables based on the access heat of the current sorted string table SSTable, and calculates the corresponding average heat frequency and the relative heat of each sorted string table SSTable to achieve compression based on access heat;

[0096] The element dequeue judgment module based on access heat calculates the heat and hotness of each sorted string table SSTable in the level when the number of KVs key-value pairs stored in the level exceeds the first preset threshold. sum , and each sorted string table SSTable and its corresponding heat and Hot sum Add to Heat and Hot sum Sorted priority queue Q, heat and Hot sum The sorting is in descending order, and the heat and Hot sum The sort string table SSTable at the head of the sorting team is dequeued;

[0097] The index construction weighing module based on access popularity determines whether there is a learned index Learned Index for the sorted string table SSTable of each dequeued element. If not, the preset error error and prefix length length used to construct the learned index Learned Index are determined based on the access popularity, and the learned index Learned Index is constructed; if so, exit and wait for the next round of judgment.

[0098] In summary, this embodiment aims to directly optimize the search process in the storage system, integrate a new learning index under read-write intensive workloads, and first calculate the access heat of the current sort string table SSTable based on positive heat, negative heat, birth timestamp and logical timestamp; then, based on the access heat of the current sort string table SSTable, calculate the average total heat of all sort string table SSTables, and calculate the corresponding average heat frequency and the relative heat of each sort string table SSTable to achieve compression based on access heat. Compression optimization based on access heat can effectively avoid the phenomenon of large fluctuations in Get operation performance; then, after the number of KVs key-value pairs stored in a certain level exceeds the first preset threshold, the heat and Hot of each sort string table SSTable in the level are calculated based on the access heat. sum , get the corresponding priority queue Q, and add the heat and Hot sum The sorted string table SSTable at the head of the sorted queue is dequeued; finally, the sorted string table SSTable of each dequeued element is weighed based on the access popularity of the index construction. Therefore, the present invention can effectively improve the reading performance and query performance of the storage system, and effectively avoid the waste of resources, so as to comprehensively improve the overall performance of the log structure tree storage method and the system.

[0099] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A log structure tree storage method based on access popularity and learning index, characterized in that: The following steps are involved: Step S1, calculating the access heat of the current sort string table SSTable based on the positive heat, negative heat, birth timestamp and logical timestamp; Step S2, based on the access heat of the current sort string table SSTable, calculate the average total heat of all sort string tables SSTables, and calculate the corresponding average heat frequency and the relative heat of each sort string table SSTable to achieve compression based on access heat; Step S3: When the number of KVs key-value pairs stored in the level exceeds the first preset threshold, the heat and Hot value of each sorted string table SSTable in the level are calculated. sum , and each sorted string table SSTable and its corresponding heat and Hot sum Add to Heat and Hot sum Sorted priority queue Q, heat and Hot sum The sorting is in descending order, and the heat and Hot sum The sort string table SSTable at the head of the sorting team is dequeued; Step S4, for each sorted string table SSTable of the dequeued element, determine whether there is a learned index LearnedIndex. If not, determine the preset error error and prefix length length used to construct the learned index Learned Index based on the access popularity, and construct the learned index Learned Index; if so, exit and wait for the next round of judgment.

2. The log structure tree storage method based on access popularity and learning index according to claim 1 is characterized in that: In step S1, the formula Calculate the access heat of the current sort string table SSTable get , among which, Hot p Indicates positive heat, Hot n represents the negative heat, α and β are the preset positive heat weight and negative heat weight respectively, T now Represents the logical timestamp, T Start Represents the birth timestamp.

3. The log structure tree storage method based on access popularity and learning index according to claim 2 is characterized in that: In step S1, for the newly generated sort string table SSTable, the formula Calculate the inherited heat of the compressed set corresponding to the newly generated sorted string table SSTable inherent , thereby inheriting the heat Hot inherent As the access popularity of the newly generated sort string table SSTable, size represents the number of KVs key-value pairs of the newly generated sort string table SSTable, size all Indicates the number of KVs key-value pairs in the compressed collection, Hot all Represents the sum of the access heats of all sorted string tables SSTables in the compressed set.

4. The log structure tree storage method based on access popularity and learning index according to claim 3 is characterized in that: In step S2, the formula Calculate the average total heat of all sorted string tables SSTables Hot avg , among which, Hot i =Hot get +Hot inherent , Hot i Represents the sum of the Get operation heat and the inheritance heat of the i-th sorted string table SSTable in level level, and n represents the number of all sorted string table SSTables.

5. The log structure tree storage method based on access popularity and learning index according to claim 4 is characterized in that: In step S2, the formula Calculate the average heat frequency Tp i , and by the formula Average heat frequency Tp i Perform heat normalization to obtain the normalized heat frequency Tp avg .

6. The log structure tree storage method based on access popularity and learning index according to claim 5 is characterized in that: In step S2, the formula Calculate the relative heat of each sort string table SSTable level-i , select relative heat Hot level-i The smallest sorted string table SSTable is used as the string table to be compressed.

7. The log structure tree storage method based on access popularity and learning index according to any one of claims 3 to 6, characterized in that: In step S3, the formula Hot sum =γ*Hot get +δ*Hot inherent Calculate the heat and hotness of each sorted string table SSTable in level level sum , where γ and δ are the preset access heat weight and inherited heat weight respectively.

8. The log structure tree storage method based on access popularity and learning index according to any one of claims 1 to 6, characterized in that: In the step S4, it is determined whether the access heat of the sorted string table SSTable is greater than the second preset threshold value. If so, the preset error value of the learned index Learned Index error is reduced and its prefix length length is increased; if not, the preset error value of the learned index Learned Index error is increased and its prefix length length is reduced; the process of constructing the learned index Learned Index is: first, assume that the elements of the selected training data set D are tuple type D i =(K i ,P i ), D i represents the i-th tuple in the training data set D, K i and P i Respectively represent the i-th key key and the corresponding logical address in the structure data SSTable to be trained; then through the formula S(K i )=P i ±error trains the interpolation point model S, where error represents the preset error; finally, the prefix search array and interpolation points are combined for query.

9. The log structure tree storage method based on access popularity and learning index according to claim 8 is characterized in that: The query process combining the prefix search array and the interpolation point is as follows: access the pointer corresponding to the logical address and prefix in the prefix search array. If the corresponding interpolation point P1 is found, then start from the interpolation point P1 and gradually traverse the interpolation point sequence backwards to find the last one that is not greater than the Get operation index Key. get interpolation point P2; otherwise, end the query; after finding the interpolation point P2, use the interpolation point P2 and the next interpolation point P3 next to the interpolation point P2 as two base points, perform interpolation operations on these two base points, obtain the predicted logical position Position, and access the index Key corresponding to the predicted logical position Position p , if Key p =Key get , the query ends; if Key p ≠Key get , then use binary search to query in the range of (Position-erroe,Position+error).

10. A log structure tree storage system based on access popularity and learning index, characterized in that: The log structure tree storage method based on access popularity and learning index as claimed in any one of claims 1 to 9 is adopted, and includes: The access heat calculation module calculates the access heat of the current sort string table SSTable based on positive heat, negative heat, birth timestamp and logical timestamp; The compression optimization module based on access heat calculates the average total heat of all sorted string table SSTables based on the access heat of the current sorted string table SSTable, and calculates the corresponding average heat frequency and the relative heat of each sorted string table SSTable to achieve compression based on access heat; The element dequeue judgment module based on access heat calculates the heat and hotness of each sorted string table SSTable in the level when the number of KVs key-value pairs stored in the level exceeds the first preset threshold. sum , and each sorted string table SSTable and its corresponding heat and Hot sum Add to Heat and Hot sum Sorted priority queue Q, heat and Hot sum The sorting is in descending order, and the heat and Hot sum The sort string table SSTable at the head of the sorting team is dequeued; The index construction weighing module based on access popularity determines whether there is a learned index Learned Index for each sorted string table SSTable of the dequeued element. If not, the preset error error and prefix length length used to construct the learned index LearnedIndex are determined based on the access popularity, and the learned index Learned Index is constructed; if so, exit and wait for the next round of judgment.

Images