How to operate the learning index tree graph

Abstract

The embodiment of the present application discloses a method for operating a learning index tree graph. The method is applied to a key-value separation storage system, wherein the key-value separation storage system is newly provided with a learning index tree graph, each node in the tree graph contains a linear model, a common prefix and an entry list, the entry list is composed of an empty entry, a data entry and a node entry, the data entry stores the key suffix and the corresponding value in the value log area of ​​the key-value separation storage system, and the node entry is associated with a child node. The learning index tree graph is constructed according to a predetermined process. After construction, the instruction content in the operation instruction can be processed; according to the instruction content, the learning index tree graph is added, deleted, modified and checked, including point query, range query, addition, deletion and modification operations. The present application enables the index mechanism of the key-value separation storage system to have the following technical effects at the same time: first, it has the ability of precise positioning; second, it has the ability of incremental update; third, it has the ability to process indefinite-length keys.

Description

Technical Field

[0001] The present application belongs to the field of computer storage technology, and in particular, relates to an operation method of a learning index tree graph. Background Art

[0002] In the era of big data, social media, email services, video sites and other applications have generated massive amounts of unstructured data, posing new challenges to storage systems. In the prior art, the values ​​in the key-value pair are decoupled from the log-structured merge tree and stored in a separate value log area, while only the address information of the key and value is stored in the log-structured merge tree; the merge operation in the log-structured merge tree no longer needs to repeatedly write the larger value part of the key-value pair, which can not only reduce the read-write amplification caused by the merge operation in the key-value storage, providing users with better read-write performance, but also reduce the amount of data written to the solid-state drive, extending the performance of the underlying solid-state drive.

[0003] Although the introduction of primary and secondary indexes can improve the read performance of key-value separation storage, these index mechanisms are traditionally implemented based on B-tree (Balanced Tree, B-tree) or its variant B+ tree. Any internal node in the tree graph needs to store multiple keys in order. When searching for the primary key of the primary index or the secondary key of the secondary index (collectively referred to as the query key), it is necessary to compare the query key from the root node to the leaf node of the tree graph from top to bottom to obtain the address information of the corresponding value in the value log area. When processing a certain internal node, it is necessary to compare or binary search all the keys stored in the node for the query key in order, which limits the search performance of the index mechanism. The ideal index mechanism needs to directly locate the position corresponding to the query key in the internal node of the tree graph. This goal can be achieved through learning indexing. Learning indexing is essentially to build a machine learning model, and call the model by inputting the query key to directly predict the position of the query key in the internal node of the tree graph.

[0004] A learning index that is highly adaptable to a key-value separation storage system needs to meet the following three important requirements: First, it must have precise positioning capabilities, that is, the constructed machine learning model has no errors, and the model prediction can directly hit the position of the query key in the internal node of the tree graph, without the need to compare the key with other positions; second, it must have incremental update capabilities, that is, when the key-value pair is updated, only the index structure needs to be adjusted locally without rebuilding the index structure; third, it must have the ability to process indefinite-length keys, that is, it supports key data to be stored in byte arrays of arbitrary length. At present, a variety of learning indexing methods have emerged at home and abroad, but none of them can meet the above three requirements at the same time. For example, RSS and SIndex do not have precise positioning capabilities, while PGM, ALEX and LIPP only support fixed-length basic data types and do not have the ability to process indefinite-length keys. Therefore, how to introduce a more powerful indexing mechanism to optimize the performance of the key-value separation storage system is a technical problem that needs to be solved urgently by technicians in this field.

[0005] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the invention

[0006] Based on this, it is necessary to propose a learning index tree graph operation method to address the above problems, which can effectively optimize the key-value separation storage system.

[0007] The present application solves the technical problem by adopting the following technical solutions:

[0008] The present application provides an operation method of a learning index tree graph, which is applied to a key-value separation storage system. The key-value separation storage system includes a log-structured merge tree and a value log area, which are used to store key-value data pairs. The key-value data pairs include keys and values, the keys and values ​​correspond one to one, and the data type of the keys is a byte array; the value log area is used to store values; the log-structured merge tree is used to store disk address pointers of keys and corresponding values; the key-value separation storage system adds a learning index tree graph; the learning index tree graph includes multiple nodes; each node is composed of a linear model, a common prefix and an entry list; the linear model is represented as: M(x)=floor(ax+b), x is the input of the linear model, a and b are model parameters, and floor(x) returns the maximum integer not greater than x; the common prefix is ​​used to represent the common part of all keys covered in the subtree with the corresponding node as the root; the entry list is used to store at least one entry, and the entries in the entry list The number of entries is between the preset lower limit min and upper limit max of the number of entries; the entry types include node entries, data entries and empty entries; the node entry is used to save the memory address pointer pointing to the lower child node of the node; the data entry is used to save the suffix of the key and the disk address pointer of the corresponding value of the corresponding key in the value log area; the empty entry does not store data and is used to be replaced by a node entry or a data entry according to the operation; the operation method of the learning index tree graph includes: performing a construction operation to construct a learning index tree graph, the construction operation includes: determining a conversion function F according to all keys stored in the key-value separation storage system, the conversion function F is a monotonic conversion function, and the monotonic conversion function is used to convert the key data of the key from a variable-length byte array to a floating-point number; determining the root node of the key-value separation storage system, and setting the common prefix of the root node to an empty byte array; constructing an input set {key1, ..., key2, ..., key3, ..., key4, ..., key5, ..., key6, ..., key7, ..., key8, ..., key9, ..., key10, ..., key11, ..., key12, ..., key13, ..., key14, ..., key15, ..., key16, ..., key17, ..., key18, ..., key19 ... N}, N is the number of keys of all keys, key i is the key, i∈[1,N], the keys in the input set are arranged in ascending order according to the corresponding byte array; the number of entries n of the root node is determined according to the number of keys N, the lower limit of the number of entries min and the upper limit of the number of entries max; the input set is used as the input of the conversion function F, and according to the first processing result F(key i ) constructs a linear model M; takes the first processing result corresponding to each key as the input of the linear model M, and substitutes it for processing to obtain the corresponding second processing result M(F(key i )); determine the entry type of the corresponding entry according to the second processing result, summarize all entries and the corresponding entry types to determine the entry list; summarize all nodes that have determined the linear model, common prefix and entry list to obtain a learning index tree graph; perform add, delete, modify and query operations on the learning index tree graph according to the received instruction content, and the add, delete, modify and query operations include point query, range query, add operation, delete operation and modify operation.

[0009] In an optional embodiment of the present application, the conversion function F is determined according to all the keys stored in the key-value separation storage system, including: obtaining all the keys key stored in the key-value separation storage system i The byte array a[1:m] is a[1:m]; the first key number and the second key number are determined according to the byte array; the first key number is defined as the number of keys whose prefix H function value is equal to H(a[1:k]) in all key data, expressed as N H(a[1:k]) ; The second key number is defined as the number of keys whose k+1th byte is equal to b among all key data whose prefix H function value is equal to H(a[1:k]), expressed as N H(a[1:k]),b ; k is the number of bytes; H is a preset function that maps from a character array to a non-negative number; the byte conditional probability P(b|H(a[1:k])) is determined based on the first key number and the second key number, and the calculation method is:

[0010] ;

[0011] The byte array probability P(a[1:k]) is defined according to the byte conditional probability iteration as:

[0012] P(a[1:0])=1; and for any k≥0, P(a[1:k+1])=P(a[1:k])·P(a[k+1]|H(a[1:k])); the conversion function F is iteratively defined based on the byte conditional probability P(b|H(a[1:k])) and the byte array probability P(a[1:k]) as:

[0013] F(a[1:0])=0; and for any k≥0, F(a[1:k+1])=F(a[1:k])+P(a[1:k])∑ b∈{0,1,…,a[k+1]-1} P(b|H(a[1:k])).

[0014] In an optional embodiment of the present application, the number of entries n of the root node is determined according to the number of keys N, the lower limit of the number of entries min and the upper limit of the number of entries max, including: if the number of keys N is less than the lower limit of the number of entries min, that is, N<min, then the number of entries n is set to the lower limit of the number of entries min, that is, n=min; if the number of keys N is greater than twice the upper limit of the number of entries max, that is, 2N>max, then the number of entries n is set to the upper limit of the number of entries max, that is, n=max; if the above situation is not the case, then the number of entries n is set to twice the number of keys N, that is, n=2N.

[0015] In an optional embodiment of the present application, according to the first processing result F(key i ) constructs a linear model M, including: calculating model parameters according to a preset lower bound q1 of the item number and a preset upper bound q2 of the item number, the calculation process is:

[0016] ;

[0017] ;

[0018] Wherein, and belong to the first processing result F(key i ); 0 ≤ q1 < q2 ≤ n - 1.

[0019] In an alternative embodiment of the present application, the entry type of the corresponding entry is determined according to the second processing result M(F(key i )) and includes: assuming that M(F(key0)) < M(F(key1)) and M(F(key N )) < M(F(key N+1 )) is always true, all second processing results M(F(key i )) are traversed; if there exists an i such that M(F(key i-1 )) < M(F(key i )) < M(F(key i+1 )), then the entry with the serial number M(F(key i )) is determined as a data entry, and the data entry is used to store the suffix of key i and the disk address pointer of the corresponding value of keyi in the value log area, and the suffix of key i is key i itself; if there exist i < j such that M(F(key i-1 )) < M(F(key i )) = M(F(key i+1 )) = … = M(F(key j )) < M(F(key j+1 )), then the new node construction operation is recursively called with {key i , …, key j} as the input, and the entry with the serial number M(F(key i )) is determined as a node entry, and the node entry is used to store the memory address pointer pointing to the new node; the entry with the serial number not equal to any M(F(key i )) is determined as an empty entry.

[0020] In an optional embodiment of the present application, a new node construction operation includes: recording the new node as node, obtaining the input keys required to construct node, and setting the common prefix of node to the longest common prefix of all input keys; determining the number of entries of node according to the number of input keys, the lower limit min of the number of entries, and the upper limit max of the number of entries, and the method for determining the number of entries of node is consistent with the method for determining the number of entries of the root node; trimming the common prefix of node from all input keys, and substituting them into the conversion function F for calculation to determine the linear model of node, and the method for determining the linear model of node is consistent with the method for determining the linear model of the root node; determining multiple entries of node according to the input keys and the linear model of node, and the method for determining the entry types of entries in node is consistent with the method for determining the entry types of entries in the root node; when an entry of node is determined to be a data entry, determining the byte array obtained by trimming the common prefix of node from the corresponding input key as the suffix of the data entry; when an entry of node is determined to be a node entry, recursively calling the new node construction operation again, and any input key during the call is the byte array obtained by trimming the common prefix of node.

[0021] In an optional embodiment of the present application, a point query includes: determining a target key to be queried in the instruction content of the point query, returning locate(root node, target key), wherein the first parameter node of the locate(node, key) function is a node in the learning index tree graph, and the second parameter key is a byte array to be queried, and the locate(node, key) function substitutes the key into the linear model M of the node for processing to obtain the target node serial number M(F(key)); determining the entry corresponding to the target node serial number M(F(key)), which is called the query entry; determining the entry type of the query entry; when the entry type of the query entry is an empty entry, marking the query entry as a positioning entry for the point query and returning information that the target key cannot be found; when the entry type of the query entry is a data entry, marking the query entry as a positioning entry for the point query, obtaining the common prefix of all nodes in the path from the root node to the node where the query entry is located, which is called the retrieval prefix, with and the suffix stored in the query entry, called the search suffix, and the result of concatenating all the search prefixes and search suffixes is marked as the comparison key; if the comparison key is not equal to the target key, the information that the target key cannot be found is returned; if the comparison key is equal to the target key, the corresponding value pointed to by the disk address pointer of the query entry stored in the value log area is returned; when the entry type of the query entry is a node entry, the child node is accessed through the memory address pointer of the lower-level child node stored in the query entry. If the common prefix of the child node is also the prefix of the key, the common prefix of the child node is cut off from the key, and the remaining part of the key is recorded as the key rem, and returns locate(child node, key rem ), otherwise the query entry is marked as a positioning entry of a point query and the information that the target key cannot be found is returned; range query, including: determining the range lower limit query key and range upper limit query key to be queried in the instruction content of the range query; executing the point query according to the range lower limit query key, determining the positioning entry of the point query, and marking it as the start entry; executing the point query according to the range upper limit query key, determining the positioning entry of the point query, and marking it as the end entry; traversing the data entries in all nodes between the start entry and the end entry in a depth-first manner; for each data entry, obtaining the common prefix of all nodes in the path from the root node to the node where the data entry is located, which is called the search prefix, and the suffix stored in the data entry, which is called the search suffix, and marking the result of splicing all the search prefixes and search suffixes as the corresponding key of the data entry; returning the corresponding keys of all data entries and the corresponding values ​​pointed to by the disk address pointer of the value log area stored therein.

[0022] In an optional embodiment of the present application, the add operation includes: determining the disk address pointer of the new key to be added in the instruction content of the add operation and the value of the new key in the value log area, recording the new key as key new , key new The disk address pointer of the value in the value log area is recorded as point new ; for key new Execute a point query to determine the location entry of the point query, which is called the first target entry; determine each node in the path from the root node to the first target entry, and mark it as a path node; concatenate the common prefixes of all path nodes and mark the result as prefix; and set the key new The result after trimming off the prefix is ​​marked as suffix new ; Determine the entry type of the first target entry; if the first target entry is an empty entry, modify the entry type of the first target entry to a data entry, and store suffix in the first target entry new and point new If the first target entry is a data entry, then the original suffix of the first target entry is obtained, recorded as suffix old ; if suffix old with suffix new If the first target entry is consistent, the log area disk address pointer is changed to point new , otherwise {suffix new ,suffix old} is used as input to call the new node construction operation to construct the first new node, which is the subtree with the root and suffix new The corresponding data entry stores suffixnew and point new , and change the first target entry to a node entry for storing a memory address pointer pointing to the first new node; if the first target entry is a node entry, mark the child node pointed to by the first target entry as an invalid node; all remaining keys covered in the subtree with the invalid node as the root and suffix new The new node construction operation is called as input to construct a second new node, and the second new node is the subtree with suffix in the root. new The corresponding data entry stores suffix new and point new , each of the remaining keys is formed by the concatenation of the common prefix of all nodes in the path from the failed node to a data entry and the suffix of the data entry; the child node pointer stored in the first target entry is replaced by the pointer of the second new node, and the failed node is deleted; it is determined whether each path node meets the preset reconstruction condition, and the reconstruction condition is that the total number of nodes in the subtree where the path node is located is not less than the preset node number threshold, nor less than the preset multiple of the number of entries in the entry list of the path node, and the proportion of data entries in all entries covered by the subtree where the path node is located is not less than the preset proportion; when the path node meets the reconstruction condition, the path node is marked as a reconstruction node; the new node construction operation is called with the set of all keys covered in the subtree with the reconstruction node as the root as input to construct a third new node, each of the remaining keys is formed by the concatenation of the common prefix of all nodes in the path from the reconstruction node to a data entry and the suffix of the data entry; the child node pointer stored in the node entry pointing to the reconstruction node is replaced by the pointer of the third new node, and the reconstruction node is deleted.

[0023] In an optional embodiment of the present application, the deletion operation includes: determining a deletion key to be deleted in the instruction content of the deletion operation, and recording the deletion key as key old , for key old Execute the point query to determine the location entry of the point query, which is called the second target entry; if the point query returns key old If no information is found, the deletion operation is terminated; otherwise, the entry type of the second target entry is changed to a null entry.

[0024] In an optional embodiment of the present application, the modification operation includes: determining the original key before modification, the new key after modification and the disk address pointer of the value of the new key in the value log area in the instruction content of the modification operation, recording the original key before modification as key old , record the modified new key as key alter , key alter The disk address pointer of the value in the value log area is recorded as point alter ; If key oldWith key alter If they are consistent, then the key alter Execute the point query to determine the location entry of the point query, which is called the third target entry; if the point query returns key alter If the information is not found, the key alter and point alter Perform the add operation as the instruction content; otherwise, change the value log area disk address pointer stored in the third target entry to point alter ; If keyold and key alter If they are not the same, first use key alter and point alter As the instruction content, perform the add operation, and then use key old Executes a delete operation as the instruction content.

[0025] The embodiments of the present application have the following beneficial effects:

[0026] The method provided in the present application can simultaneously have the following three advantages for the database: first, it has the ability of precise positioning, that is, the constructed machine learning model has no errors, and the model prediction can directly hit the position of the query key in the internal node of the tree graph, without the need to compare the key with other positions; second, it has the ability of incremental update, that is, when the key-value pair is updated, only the index structure needs to be locally adjusted without rebuilding the index structure; third, it has the ability to process indefinite-length keys, that is, it supports the storage of key data in byte arrays of arbitrary length.

[0027] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented according to the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with the accompanying drawings. It should be understood that the above general description and the detailed description below are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] in:

[0030] Figure 1 A schematic flow chart of an operation method of a learning index tree graph provided by an embodiment;

[0031] Figure 2 A schematic diagram of the architecture of an existing key-value separation storage system provided by an embodiment;

[0032] Figure 3 A schematic diagram of the architecture of a newly added learning index tree graph provided by an embodiment;

[0033] Figure 4 A schematic block diagram of the structure of a computer device provided by an embodiment. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] To clearly describe the operation method of a learning index tree graph provided in this embodiment, please refer to Figure 1~Figure 3 .

[0036] In the era of big data, social media, email services, video sites and other applications have generated massive amounts of unstructured data, posing new challenges to storage systems. Key-value storage based on Log-Structure Merge-Tree (LSM-tree) can efficiently store and access massive amounts of unstructured data, and has thus become the core technology for large-scale storage in modern data centers. However, with the explosive growth of data, the application of new storage hardware such as solid-state drives, and the emergence of application scenarios such as primary and secondary indexes have made the storage environment increasingly complex. Traditional key-value storage based on log-structured merge-trees is limited by its own architecture and is difficult to provide better performance for upper-layer applications. There are two problems.

[0037] On the one hand, traditional key-value storage does not optimize read and write performance for SSDs. Key-value storage based on log-structured merge trees uses a multi-layer data structure to manage stored data - appending data in the form of log writes and finally persisting it to the storage device. When the remaining space of the higher layer receiving the latest written data is small, the log-structured merge tree needs to call a compaction operation to reclaim the storage space occupied by the discarded data and write the legal data to the lower layer with larger space. Frequent compaction operations will rewrite a large amount of legal data, causing additional write amplification, which will not only seriously reduce the performance of key-value storage, but also greatly reduce the service life of the underlying SSD. In addition, as the amount of data grows, the key-value pairs are scattered in different layers roughly according to the order of write time. Read operations need to search layer by layer to obtain the corresponding key-value pairs, causing additional read amplification and restricting read performance.

[0038] On the other hand, traditional key-value storage is not optimized for the application scenarios of primary and secondary indexes, and it is difficult to provide efficient search performance for secondary indexes. In traditional key-value storage, primary and secondary indexes are also subject to the inefficiency caused by the data organization structure and merge operation of the log structure merge tree, and additional optimization is required to release better performance. In addition, there is an index navigation problem with secondary indexes. The index navigation problem means that in the secondary index, the auxiliary attributes to be indexed are used as keys, and the primary key is used as the value storage of the secondary index; when a search operation is issued to the secondary index, the primary key is first obtained by searching the secondary index, and then the corresponding value is found in the primary index through the primary key; therefore, the search operation on the secondary index requires an additional query operation.

[0039] The idea of ​​key-value separation provides a feasible framework for efficient use of solid-state drives and optimization of primary and secondary indexes for key-value storage based on log-structured merge trees. The architectures of mainstream key-value separation storage systems such as WiscKey, SineKV, and DiffKV are shown in the following figure. Figure 2As shown. These systems decouple the values ​​in the key-value pair from the log-structured merge tree and store them in a separate value log area, while only the address information of the key and value is stored in the log-structured merge tree; the merge operation in the log-structured merge tree no longer needs to repeatedly write the larger value part of the key-value pair, which can not only reduce the read-write amplification caused by the merge operation in the key-value storage and provide users with better read-write performance, but also reduce the amount of data written to the solid-state drive and extend the performance of the underlying solid-state drive. In addition, key-value separation is also suitable for solving index navigation problems. For a key-value pair, the value is stored in a separate value log area, and the primary index and the secondary index only store the primary key or secondary key, as well as the address information of the value in the value log area. When a query operation is issued to the secondary index, the corresponding value can be directly obtained without querying the primary index again, thus avoiding the overhead caused by index navigation.

[0040] Although the introduction of primary and secondary indexes can improve the read performance of key-value separation storage, these index mechanisms are traditionally implemented based on B-tree (Balanced Tree, B-tree) or its variant B+ tree. Any internal node in the tree graph needs to store multiple keys in order. When searching for the primary key of the primary index or the secondary key of the secondary index (collectively referred to as the query key), it is necessary to compare the query key from the root node to the leaf node of the tree graph from top to bottom to obtain the address information of the corresponding value in the value log area. When processing a certain internal node, it is necessary to compare or binary search all the keys stored in the node for the query key in order, which limits the search performance of the index mechanism. The ideal index mechanism needs to directly locate the position corresponding to the query key in the internal node of the tree graph. This goal can be achieved through learning indexing. Learning indexing is essentially to build a machine learning model, and call the model by inputting the query key to directly predict the position of the query key in the internal node of the tree graph. A learning index that is highly adaptable to a key-value separation storage system needs to meet the following three important requirements: First, it must have accurate positioning capabilities, that is, the constructed machine learning model has no errors, and the model prediction can directly hit the position of the query key in the internal node of the tree graph without the need to compare the key with other positions; second, it must have incremental update capabilities, that is, when the key-value pair is updated, only the index structure needs to be adjusted locally without rebuilding the index structure; third, it must have the ability to process indefinite-length keys, that is, it supports key data to be stored in byte arrays of arbitrary length. At present, a variety of learning indexing methods have emerged at home and abroad, but none of them can meet the above three requirements at the same time. For example, RSS and SIndex do not have accurate positioning capabilities, while PGM, ALEX, and LIPP only support fixed-length basic data types and do not have the ability to process indefinite-length keys.

[0041] In order to further improve the read performance of the key-value separation storage system, the industry currently urgently needs a learning indexing method that meets the above three requirements. Based on the above technical problems and technical difficulties, this application proposes a learning index tree graph operation method that can meet the above requirements at the same time.

[0042] The method proposed in this application is applied to a key-value separation storage system. The specific architecture of the key-value separation storage system is as follows Figure 2 As shown, including the log structure merge tree and the value log area, the key-value separation storage system is used to store key-value data pairs. Specifically, the key-value data pair includes a key and a value, the key and the value correspond one to one, the data type of the key is a byte array, and the data type of the value is not limited.

[0043] The value log area is used to store values; the log structure merge tree is used to store keys and disk address pointers of corresponding values.

[0044] The innovation of this application is that, in this application, the key-value separation storage system adds a learning index tree graph. For the specific architecture of the learning index tree graph newly added in this application, please refer to Figure 3 .like Figure 3 As shown, the learned index tree diagram includes multiple nodes. Each node consists of a linear model, a common prefix and an entry list.

[0045] The linear model is expressed as: M(x)=floor(ax+b), where x is the input of the linear model, a and b are model parameters, and floor(x) returns the largest integer not greater than x.

[0046] The common prefix is ​​used to represent the common part of all keys covered in the subtree rooted at the corresponding node.

[0047] The entry list is used to store at least one entry, and the number of entries in the entry list is between a preset lower limit of the number of entries min and an upper limit of the number of entries max. The entry types include node entries, data entries, and empty entries.

[0048] Specifically, the difference between the three types of entries is that the node entry is used to store the memory address pointer to the lower-level child node of the node. The data entry is used to store the key suffix and the disk address pointer of the corresponding value of the corresponding key in the value log area. The empty entry does not store data and is used to be replaced by the node entry or data entry according to the operation.

[0049] For the newly added learning index tree graph in the present application, an operation method of the learning index tree graph is actually configured, which specifically includes steps S110 to S120.

[0050] Step S110: Execute a construction operation to construct a learning index tree graph, the construction operation comprising: determining a conversion function F according to all keys stored in the key-value separation storage system, the conversion function F being a monotonic conversion function, the monotonic conversion function being used to convert the key data of a key from an indefinite-length byte array to a floating-point number; determining a root node of the key-value separation storage system, setting the common prefix of the root node to an empty byte array; constructing an input set {key1, ..., key2, ..., key3, ..., key4, ..., key5, ..., key6, ..., key7, ..., key8, ..., key9, ..., key10, ..., key11, ..., key12, ..., key13, ..., key14, ..., key15, ..., key16, ..., key17, ..., key18, ..., key19, ..., key19, ..., key11 N}, N is the number of keys of all keys, key i is the key, i∈[1,N], the keys in the input set are arranged in ascending order according to the corresponding byte array; the number of entries n of the root node is determined according to the number of keys N, the lower limit of the number of entries min and the upper limit of the number of entries max; the input set is used as the input of the conversion function F, and according to the first processing result F(key i ) constructs a linear model M; the first processing result corresponding to each key is used as the input of the linear model M, and the corresponding second processing result M(F(key i )); determine the entry type of the corresponding entry according to the second processing result, summarize all entries and the corresponding entry types to determine the entry list; summarize all nodes that have determined the linear model, common prefix and entry list to obtain a learning index tree diagram.

[0051] In one embodiment, step S110 is actually performed when the key-value separation storage system needs to create or rebuild the learning index. First, the conversion function F of the key data, ie, the byte array to the floating point number, is determined according to all the keys currently stored in the system.

[0052] In one implementation, step S110: determining a conversion function F according to all keys stored in the key-value separation storage system, specifically includes the following steps.

[0053] First, get all the keys stored in the key-value separation storage system i The byte array a[1:m]. The first key number and the second key number are determined based on the byte array. The two are specifically defined as: The first key number is defined as the number of keys whose prefix H function value is equal to H(a[1:k]) in all key data, expressed as N H(a[1:k]) ; The second key number is defined as the number of keys whose k+1th byte is equal to b among all key data whose prefix H function value is equal to H(a[1:k]), expressed as N H(a[1:k]),b . Wherein, k is the number of bytes; H is a preset function that maps from a character array to a non-negative number. This application does not make further limitations on the mapping function H, but only gives a specific calculation formula as an example below. Changing the calculation method should not be considered to exceed the technical protection scope of this application.

[0054] Furthermore, the byte conditional probability P(b|H(a[1:k])) is determined according to the first key number and the second key number. The calculation method can refer to:

[0055] (1)

[0056] The byte array probability P(a[1:k]) is defined according to the byte conditional probability iteration as:

[0057] P(a[1:0])=1, and for any k≥0, P(a[1:k+1])=P(a[1:k])·P(a[k+1]|H(a[1:k])).

[0058] The transformation function F is iteratively defined based on the byte conditional probability P(b|H(a[1:k])) and the byte array probability P(a[1:k]) as:

[0059] F(a[1:0])=0, and for any k≥0, F(a[1:k+1])=F(a[1:k])+P(a[1:k])∑ b∈{0,1,…,a[k+1]-1} P(b|H(a[1:k])).

[0060] Therefore, it can be proved that the transfer function F is a monotonic transfer function. Assume that the value range of H function is 0 to N max -1, then for all different H(a[1:k]) and b, N H(a[1:k]) and N H(a[1:k]),b All of them can be obtained by traversing all the key data stored in the current system, where N H(a[1:k]) Occupancy N max The storage space for floating point numbers, N H(a[1:k]),b Occupies 256N max In a specific embodiment, H(a[1:k]) is defined as the function value of the base64 encoded string c[1:m] of a[1:k], 4096·c[1]+ 64·c[floor((m+1) / 2)] + c[m], where each c[i] character is considered to be a value between 0 and 63, and when m<3, c[1:m] is padded with null characters corresponding to 0 to make up 3 characters; the N corresponding to this H function max =262144.

[0061] The following proves that the conversion function F is a monotonic conversion function: For any character arrays a1[1:m] and a2[1:n] that satisfy a1 < a2, there must exist a sequence number k ≥ 1 such that a1[1:k - 1] = a2[1:k - 1] and 0 ≤ a1[k] < a2[k] ≤ 255. Here, for any k > m, a1[k] is defined as 0, and for any k > n, a2[k] is defined as 0. Denote the byte array obtained by concatenating a1[1:p] with 256 as a3[1:p + 1], that is, a3[1:p] = a1[1:p] and a3[p + 1] = 256. Then we have:

[0062] F(a3[1:k + 1]) = F(a1[1:k]) + P(a1[1:k])

[0063] = F(a1[1:k - 1]) + P(a1[1:k - 1])∑ b∈{0,…,a1[k]-1} P(b|H(a[1:k - 1])) + P(a1[1:k - 1])P(a1[k]|H(a1[1:k - 1]))

[0064] = F(a2[1:k - 1]) + P(a2[1:k - 1])∑ b∈{0,…,a1[k]} P(b|H(a[1:k - 1]))

[0065] <F(a2[1:k - 1]) + P(a2[1:k - 1])∑ b∈{0,…,a2[k]-1} P(b|H(a[1:k - 1]))

[0066] = F(a2[1:k]).

[0067] Similarly, it can be proved that F(a3[1:k + 2]) < F(a3[1:k + 1]), ……, F(a3[1:m]) < F(a3[1:m - 1]). Therefore, F(a1[1:m]) < F(a3[1:m]) < …… < F(a3[1:k + 1]) < F(a2[1:k]) ≤ F(a2[1:n]), from which it can be seen that F is a monotonic conversion function.

[0068] This application aims to construct a simple and efficient key - value pair position prediction model. It is necessary to convert the byte array into a floating - point number so that the prediction model can take the floating - point number as input. In addition, to ensure the efficiency of range queries, that is, to ensure that all key sets between the target entry corresponding to the range upper limit and the target entry corresponding to the range lower limit are the exact results of the range query, it is necessary to ensure that the conversion function from the byte array to the floating - point number is monotonic. A relatively simple and intuitive conversion function G is defined as:

[0069] G(a[1:m]) = a[1] / 256 + a[2] / 256 2+ … + a[m] / 256 m 。

[0070] It can be proved that this conversion function is also monotonic, that is, a1[1:m] < a2[1:n] means G(a1[1:m]) < G(a2[1:n]). This conversion function is equivalent to fixing the conditional probability P(b|H(a[1:k])) used in the conversion function F proposed in this application to 1 / 256. Using a fixed conditional probability will cause the learning-based index method to locate a large number of similar key data to the same position, resulting in the need for a deeper subtree to distinguish different key data at the constructed new node at this position. This application determines the corresponding conditional probability according to the key distribution of the current system, and can distribute similar key data to different positions as much as possible, thereby reducing the depth of the subtree at these positions. That is to say, compared with the conversion function G, the byte array conversion function F proposed in this application can make the learned tree graph index structure shallower, thereby improving the query efficiency.

[0071] After determining F, call the construction process of the root node to construct the root node, and the root node also has a linear model, a common prefix, and an entry list. This construction process takes the input set of different key sets stored in the current system, denoted as {key1,…,key N} as input, where key1, ……, key N have been sorted in ascending order of byte arrays, and i ∈ [1, N]. The construction process will be described one by one later. First, for the common prefix, set the common prefix of the root node to an empty byte array. Secondly, for the number of entries n, it can be determined according to N, that is, the number of elements in the input set, and the preset minimum value min and maximum value max of the number of entries, so that n is between min and max.

[0072] In one embodiment, determining the number of entries n of the root node according to the number of keys N, the lower limit min of the number of entries, and the upper limit max of the number of entries includes:

[0073] If the number of keys N is less than the lower limit min of the number of entries, that is, N < min, then set the number of entries n to the lower limit min of the number of entries, that is, n = min.

[0074] If the number of keys N is greater than twice the upper limit max of the number of entries, that is, 2N > max, then set the number of entries n to the upper limit max of the number of entries, that is, n = max.

[0075] If it is not the above situation, the number of entries n is set to twice the number of keys N, that is, n = 2N. The above construction method is only a preferred embodiment, and in other embodiments, improvements can be made with reference to the above method, and all improvements are not considered to exceed the scope provided by this application. The actual purpose of determining the number of entries n of the root node is to ensure that there are enough empty entries in the entry list stored in the root node to accommodate the keys added later.

[0076] This application limits the size of the entry list maintained in each node of the learning index between a preset minimum value min and a maximum value max, so that the entire entry list can be processed using a complete memory page, improving processing efficiency. That is to say, only need to set max to the maximum number of entries that a single memory page can accommodate, then even if the number of keys input when constructing a node exceeds the number of entries that a single memory page can accommodate, the entry list of this node can be stored in a single memory page. In addition, when constructing the entry list of a node in the technical solution of this application, some empty entries will be reserved for the entry list according to the preset min and max, so that when new keys are added later, they can be quickly inserted into the empty entries, and try to avoid merging with the original keys of the data entries where the new keys are located to construct a new node.

[0077] After determining the common prefix of the root node and the number of entries n of the entry list, the linear model can be determined.

[0078] In one embodiment, a linear model M is constructed according to the first processing result F(key i ), including: calculating model parameters according to a preset lower bound q1 of the entry number and a preset upper bound q2 of the entry number, and the calculation process can be:

[0079] (2)

[0080] (3)

[0081] In the above formula, and belong to the first processing result F(key i ); 0 ≤ q1 < q2 ≤ n - 1. Construct the linear model M(x) = floor(ax + b) of the root node, so that M(F(key1)) and M(F(key N )) fall within the closed interval [0, n - 1]. The assignment of the above parameters a and b satisfies a > 0, M(F(key1)) is greater than or equal to 0, and M(F(key N )) is less than or equal to n - 1. Setting M(F(key1)) to be greater than but not equal to 0 can reserve the empty entry with the smallest serial number for the entry list; setting M(F(key N)) Less than and not equal to n - 1 can reserve an empty entry with the maximum serial number for the entry list. In a specific embodiment, min is set to 8, q2 is set to n - 2, q1 is set to 1, and correspondingly a = (n - 3) / (F(key N ) - F(key1)) and b = 1 - a·F(key1).

[0082] Finally, for convenience of description, fictional key0 and key are introduced N+1 and it is assumed that M(F(key0)) < M(F(key1)) and M(F(key N )) < M(F(key N+1 )) is always true. Construct an entry list with entry serial numbers starting from 0 to determine the entry type of each entry.

[0083] In an implementation manner, according to the second processing result M(F(key i )) to determine the entry type of the corresponding entry, including: assuming M(F(key0)) < M(F(key1)) and M(F(key N )) < M(F(key N+1 )) is always true, traverse all the second processing results M(F(key i ))

[0084] If there exists an i such that M(F(key i-1 )) < M(F(key i )) < M(F(key i+1 ))), then determine the entry with the serial number M(F(key i )) as a data entry, and the data entry is used to store the suffix of key i and the disk address pointer of the corresponding value of keyi in the value log area. The suffix of key i is key i itself.

[0085] If there exist i < j such that M(F(key i-1 )) < M(F(key i )) = M(F(key i+1 )) = … = M(F(key j )) < M(F(key j+1 ))), then recursively call the new node construction operation with {key i ,…,key j} as the input, and determine the entry with the serial number M(F(key i )) as a node entry, and the node entry is used to store the memory address pointer pointing to the new node.

[0086] The sequence number is not equal to any M(F(key i )) is determined to be an empty entry.

[0087] In one embodiment, a new node construction operation includes: recording the new node as node, obtaining the input keys required to construct node, and setting the common prefix of node to the longest common prefix of all input keys; determining the number of entries of node according to the number of input keys, the lower limit min of the number of entries, and the upper limit max of the number of entries, and the method for determining the number of entries of node is consistent with the method for determining the number of entries of the root node; clipping the common prefix of node from all input keys, and substituting them into the conversion function F for calculation to determine the linear model of node, and the method for determining the linear model of node is consistent with the method for determining the linear model of the root node; determining multiple entries of node according to the input keys and the linear model of node, and the method for determining the entry types of entries in node is consistent with the method for determining the entry types of entries in the root node; when an entry of node is determined to be a data entry, determining the byte array obtained by clipping the common prefix of node from the corresponding input key as the suffix of the data entry; when an entry of node is determined to be a node entry, recursively calling the new node construction operation again, and any input key during the call is the byte array obtained by clipping the common prefix of node.

[0088] In one embodiment, the construction process of the new node is similar to the construction process of the root node, which is also a recursive process using the conversion function F. The steps include determining the common prefix of the new node, determining the number of entries of the new node, and constructing a linear model and an entry list of the new node. The difference is that:

[0089] First, assume that the input key for constructing a new node node is key1[1:m1], ..., key n [1:m n ], the common prefix stored in node is set to key1[1:m1], ..., key n [1:k n ]’s longest common prefix key1[1:p].

[0090] Second, when constructing the linear model of node, first cut off the common prefix of node from all input keys and then substitute them into the transformation function F for calculation, that is, key1[p+1:m1], ..., key n [p+1:m n ]Construct a linear model.

[0091] Third, when an entry in the entry list is determined as a data entry, the corresponding input key key i The byte array obtained after trimming the common prefix of the node is the key i[p+1:m i is determined as the suffix of this entry; when determining an entry in the entry list as a node entry, assume that the input key for locating this node entry through the linear model is key i [1:m i , …, key j [1:m j , where i < j, then use {key i [p+1:m i , …, key j [p+1:m j} as the input to recursively call the construction process of the new node, and determine the memory address pointer of the constructed new node as the stored content of this entry.

[0092] Summarize all nodes for which the linear model, common prefix, and entry list are determined to obtain the learning index tree diagram.

[0093] For the newly added learning index of this application, its structure stores the information of all key data in the current system, and divides each complete key key into different parts and stores them on the path from the root node to a certain data entry. The nodes on the path store the prefixes of different parts of the key, and the data entry at the end of the path stores the suffix of the key, that is, the key is the concatenation result of these prefixes and suffixes. This storage method saves the storage space of the key because the prefix of the key shares space with the prefixes of other keys. In addition, this storage method also reflects the advantage of the learning index structure relative to the traditional prefix tree (Trie), that is, each node stores multiple bytes in the key data, while each node of the prefix tree only stores 1 byte. Therefore, based on the learning index structure, in the subsequent point query operation compared with the point query operation based on the traditional prefix tree, fewer nodes are experienced to locate the target node, thus having higher query efficiency.

[0094] Step S120: Perform addition, deletion, modification, and query operations on the learning index tree diagram according to the received instruction content. The addition, deletion, modification, and query operations include point query, range query, addition operation, deletion operation, and modification operation.

[0095] In one embodiment, the instruction is an instruction issued by the host computer or the user and used to control the key-value separation storage system. In this application, it is used to perform addition, deletion, modification, and query operations on the learning index tree diagram, specifically including point query, range query, addition operation, deletion operation, and modification operation. Subsequently, how each operation is performed will be shown one by one.

[0096] In one embodiment, the addition, deletion, modification, and query operations include point query and range query, which are used to determine the key-value pairs to be processed. Below, how the two query operations are executed will be described separately.

[0097] For point query, it includes: first determine the target key to be queried in the instruction content of the point query, locate it through the locatae function, and return the result of locate(root node, target key). Among them, the first parameter node of the locate(node, key) function is the node in the learning index tree graph, and the second parameter key is the byte array to be queried. Specifically, the locate(node, key) function substitutes key into the linear model M of node for processing, and obtains the target node number M(F(key)). Determine the entry corresponding to the target node number M(F(key)), which is called the query entry.

[0098] Determine the type of the query entry:

[0099] When the item type of the query item is an empty item, the query item is marked as a location item of a point query and the information that the target key cannot be found is returned;

[0100] When the query entry type is a data entry, mark the query entry as a positioning entry for point query, obtain the common prefix of all nodes in the path from the root node to the node where the query entry is located, called the search prefix, and the suffix stored in the query entry, called the search suffix. Mark the result of concatenating all search prefixes and search suffixes as a comparison key. Compare whether the comparison key is equal to the target key: if the comparison key is not equal to the target key, return the information that the target key cannot be found; if the comparison key is equal to the target key, return the corresponding value pointed to by the disk address pointer stored in the value log area of ​​the query entry.

[0101] When the query entry is a node entry, the child node is accessed through the memory address pointer of the lower-level child node stored in the query entry. If the common prefix of the child node is also the prefix of the key, the key is cut off from the common prefix of the child node, and the remaining part of the key is recorded as the key. rem , and returns locate(child node, key rem ). Otherwise, mark the query entry as a location entry for a point query and return the information that the target key cannot be found. Among them, locate the key in the child node rem The process is the same as the process of locating the key in the root node, that is, according to the sequence number M(F(key rem ))'s different types of entries are handled differently.

[0102] In a point query operation, locating the query key only requires accessing a certain data entry or empty entry from top to bottom along a single path starting from the root node, and each node along the way only accesses one entry, so the learning indexing method has accurate positioning capabilities.

[0103] For range query, the operation is implemented based on point query, and the specific steps include: determining the lower limit query key and upper limit query key of the range to be queried in the instruction content of the range query. Execute the point query according to the lower limit query key, determine the positioning entry of the point query, and mark it as the start entry; execute the point query according to the upper limit query key, determine the positioning entry of the point query, and mark it as the end entry. Traverse the data entries in all nodes between the start entry and the end entry in a depth-first manner. For each data entry, obtain the common prefix of all nodes in the path from the root node to the node where the data entry is located, which is called the retrieval prefix, and the suffix stored in the data entry, which is called the retrieval suffix. Mark the result of splicing all retrieval prefixes and retrieval suffixes as the corresponding key of the data entry. Return the corresponding keys of all data entries and the corresponding values ​​pointed to by the disk address pointer of the value log area stored therein.

[0104] In one embodiment, when the operation instruction is an add operation, the specific steps include: first determining the disk address pointer of the new key to be added in the instruction content of the add operation and the value of the new key in the value log area, recording the new key as key new , key new The disk address pointer of the value in the value log area is recorded as point new .

[0105] Key new Execute the point query to determine the location entry of the point query, which is called the first target entry. Determine each node in the path from the root node to the first target entry and mark it as a path node. Concatenate the common prefixes of all path nodes and mark the result as prefix. new The result after trimming off the prefix is ​​marked as suffix new .

[0106] Then, the item type of the first target item is determined, and different processing flows are executed according to different item types.

[0107] If the first target entry is an empty entry, the entry type of the first target entry is changed to a data entry, and suffix is ​​stored in the first target entry. new and point new .

[0108] If the first target entry is a data entry, then get the original suffix of the first target entry, recorded as suffix old . Determine if suffix old with suffix new If the first target entry is consistent, the log area disk address pointer is changed to point new , otherwise {suffix new,suffix old} is used as input to call the new node construction operation to construct the first new node. The new node construction operation has been described in detail in the previous text and will not be repeated here. new The corresponding data entry stores suffix new and point new , and changes the first target entry to a node entry for storing a memory address pointer pointing to the first new node.

[0109] If the first target entry is a node entry, the child node pointed to by the first target entry is marked as an invalid node. All remaining keys and suffixes covered in the subtree with the invalid node as the root new The new node construction operation is called with the set as input to construct a second new node, where each of the remaining keys is formed by concatenating the common prefix of all nodes in the path from the invalid node to a data entry and the suffix of the data entry. The second new node is the root of the subtree with suffix new The corresponding data entry stores suffix new and point new Then, the child node pointer stored in the first target entry is replaced with the pointer of the second new node, and the invalid node is deleted.

[0110] After completing the above storage process, determine whether each path node meets the preset reconstruction condition: when the path node meets the reconstruction condition, mark the path node as a reconstruction node. Call the new node construction operation with the set of all remaining keys covered in the subtree with the reconstruction node as the root as input to construct a third new node, and each of the remaining keys is composed of the common prefix of all nodes in the path from the reconstruction node to a data entry and the suffix of the data entry. Replace the child node pointer stored in the node entry pointing to the reconstruction node with the pointer of the third new node, and delete the reconstruction node. In a specific embodiment, the reconstruction condition can be that the total number of nodes in the subtree where the path node is located is not less than a preset node number threshold (such as 64), nor less than a preset multiple of the number of entries in the entry list of the path node (such as 2), and the proportion of data entries in all entries covered by the subtree where the path node is located is not less than a preset proportion (such as 10%).

[0111] The adding operation process proposed in this application requires that when the nodes in the new key adding path meet the preset reconstruction conditions, the subtree with the node as the root is reconstructed, that is, all the keys contained in the subtree are used as input to construct a new node, and the subtree with the new node as the root replaces the subtree with the node as the root. This application only gives an example of a definition framework for reconstruction conditions, and it should not be considered that the specific assignment of parameters in the definition framework or the modification of the definition framework exceeds the technical protection scope of this application.

[0112] In one embodiment, when the operation instruction is a delete operation, the specific steps include: determining the delete key to be deleted in the instruction content of the delete operation, recording the delete key as key old . old Execute the point query to determine the location entry of the point query, which is called the second target entry. old If no information is found, the deletion operation is terminated; otherwise, the entry type of the second target entry is changed to a null entry.

[0113] In one embodiment, when the operation instruction is a modification operation, the specific steps include: determining the original key before modification, the new key after modification and the disk address pointer of the value of the new key in the value log area in the instruction content of the modification operation, recording the original key before modification as key old , record the modified new key as key alter , key alter The disk address pointer of the value in the value log area is recorded as point alter .

[0114] The modification operation needs to determine whether the modified object is the corresponding value of the key or the key itself, so as to execute different modification processes.

[0115] If key old With key alter consistent, that is, the modified object is the corresponding value of the key, then the key alter Execute the point query to determine the location entry of the point query, which is called the third target entry. alter If the information is not found, the key alter and point alter Perform the add operation as the instruction content; otherwise, change the value log area disk address pointer stored in the third target entry to point alter .

[0116] If key old With key alter If the modified object is the key itself, then first use key alter and point alter As the instruction content, perform the add operation, and then use key old Executes a delete operation as the instruction content.

[0117] According to the key value addition, deletion and modification operations described above, the index structure does not need to be rebuilt when updating, so the learning indexing method has the ability of incremental update. In addition, the key data processed by the learning indexing method is a byte array of arbitrary length, which can be used to express any type of data. The present application introduces a monotonic conversion function from variable-length byte arrays to floating-point numbers, so that the subsequent processing of variable-length keys is converted into a simpler and more efficient processing of floating-point keys, and thus has the ability to effectively process variable-length keys. The learning indexing method has the same level of time complexity as the current cutting-edge learning indexing method LIPP in operations such as point query and new key addition. Specifically, the worst-case time complexity of point query is O(logN), and the amortized time complexity of new key addition is O(log 2 N), other operations can be converted into point queries and / or new key additions to determine the time complexity, where N is the number of different keys in the key-value separation storage system.

[0118] Therefore, the method provided in the present application can simultaneously possess the following three advantages for the database: first, it has the ability of precise positioning, that is, the constructed machine learning model has no errors, and the model prediction can directly hit the position of the query key in the internal node of the tree graph, without the need to compare the key with other positions; second, it has the ability of incremental update, that is, when the key-value pair is updated, only the index structure needs to be adjusted locally, and there is no need to rebuild the index structure; third, it has the ability to process indefinite-length keys, that is, it supports the storage of key data in byte arrays of arbitrary length.

[0119] Figure 4 FIG. 1 shows an internal structure diagram of a computer device in an embodiment. The computer device may be a terminal or a server. Figure 4 As shown, the computer device includes a processor, a memory and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor may implement the operating method of the learning index tree graph. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor may implement the operating method of the learning index tree graph. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] In one embodiment, the present application further proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method described in any of the aforementioned embodiments.

[0121] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0122] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for operating a learning index tree graph, applied to a key-value separation storage system, wherein the key-value separation storage system comprises a log structure merge tree and a value log area, for storing key-value data pairs, wherein the key-value data pairs comprise a key and a value, wherein the key and the value correspond one to one, and the data type of the key is a byte array; the value log area is used to store the value; and the log structure merge tree is used to store a disk address pointer of the key and the corresponding value; It is characterized in that The key-value separation storage system adds a learning index tree graph; The learning index tree graph includes a plurality of nodes; each of the nodes is composed of a linear model, a common prefix and an entry list; the linear model is expressed as: M(x)=floor(ax+b), x is the input of the linear model, a and b are model parameters, and floor(x) returns the maximum integer not greater than x; the common prefix is ​​used to represent the common part of all keys covered in the subtree with the corresponding node as the root; the entry list is used to store at least one entry, and the number of entries in the entry list is between a preset lower limit min and an upper limit max of the number of entries; The entry types of the entry include node entry, data entry and empty entry; the node entry is used to store the memory address pointer pointing to the lower-level child node of the node; the data entry is used to store the suffix of the key and the disk address pointer of the corresponding value of the key in the value log area; The empty entry does not store data and is used to be replaced by the node entry or the data entry according to an operation; The operation method of the learning index tree graph includes: Performing a construction operation to construct the learning index tree graph, the construction operation comprising: determining a conversion function F according to all the keys stored in the key-value separation storage system, the conversion function F being a monotonic conversion function, and the monotonic conversion function being used to convert key data of the key from a variable-length byte array to a floating-point number; Determine a root node of the key-value separation storage system, and set a common prefix of the root node to an empty byte array; Construct an input set {key1, ..., key N }, N is the number of keys of all the keys, key i is the key, i∈[1,N], the keys in the input set are arranged in ascending order according to the corresponding byte arrays; the number of entries n of the root node is determined according to the number of keys N, the lower limit of the number of entries min and the upper limit of the number of entries max; The input set is used as the input of the conversion function F, and the first processing result F(key i )Construct a linear model M; The first processing result corresponding to each key is used as the input of the linear model M to obtain the corresponding second processing result M(F(key i ); determining the entry type of the corresponding entry according to the second processing result, summarizing all the entries and the corresponding entry types to determine the entry list; summarizing all the nodes that determine the linear model, the common prefix and the entry list to obtain the learning index tree diagram; The learning index tree graph is subjected to add, delete, modify and query operations according to the received instruction content, and the add, delete, modify and query operations include point query, range query, add operation, delete operation and modify operation.

2. The method for operating a learning index tree graph according to claim 1, characterized in that: The step of determining a conversion function F according to all the keys stored in the key-value separation storage system includes: Get all the keys stored in the key-value separation storage system i Byte array a[1:m]; Determine the first key number and the second key number respectively according to the byte array; the first key number is defined as the number of keys whose prefix H function value in all key data is equal to H(a[1:k]), expressed as N H(a[1:k]) The second key number is defined as the number of keys whose k+1th byte is equal to b in all key data whose prefix H function value is equal to H(a[1:k]), expressed as N H(a[1:k]),b ; k is the number of bytes; H is a preset function that maps from a character array to a non-negative number; The byte conditional probability P(b|H(a[1:k])) is determined according to the first key number and the second key number, and the calculation method is: ； The byte array probability P(a[1:k]) is defined according to the byte conditional probability iteration as: P(a[1:0])=1; and for any k≥0, P(a[1:k+1])=P(a[1:k])·P(a[k+1]|H(a[1:k])); The conversion function F is iteratively defined according to the byte conditional probability P(b|H(a[1:k])) and the byte array probability P(a[1:k]) as: F(a[1:0])=0; and for any k≥0, F(a[1:k+1])=F(a[1:k])+P(a[1:k])∑ b∈{0,1,…,a[k+1]-1} P(b|H(a[1:k]))。 3. The method for operating a learning index tree graph according to claim 1, characterized in that: The determining the number of entries n of the root node according to the number of keys N, the lower limit min of the number of entries, and the upper limit max of the number of entries includes: If the number of keys N is less than the lower limit of the number of entries min, that is, N<min, the number of entries n is set to the lower limit of the number of entries min, that is, n=min; If the number of keys N is greater than twice the upper limit of the number of entries max, that is, 2N>max, then the number of entries n is set to the upper limit of the number of entries max, that is, n=max; If the above situation is not the case, the number of entries n is set to twice the number of keys N, that is, n=2N.

4. The method for operating a learning index tree graph according to claim 1, wherein: The first processing result F(key i )Construct a linear model M, including: The model parameters are calculated according to the preset lower bound q1 of the item number and the preset upper bound q2 of the item number. The calculation process is: ； ； In the formula, and Belonging to the first processing result F(key i );0≤q1 <q2≤n-1。 5. The method for operating a learning index tree graph according to claim 1, characterized in that: According to the second processing result M(F(key i )) Determine the entry type of the corresponding entry, including: Assume that M(F(key0)) < M(F(key1)) and M(F(key N )) < M(F(key N+1 )) is always true, and traverse all the second processing results M(F(key i )); If there exists i that satisfies M(F(key i-1 )) < M(F(key i )) < M(F(key i+1 )), then the sequence number is M(F(key i )) is determined as the data entry, and the data entry is used to store the key i The suffix and the key i The disk address pointer of the corresponding value in the value log area, the key i The suffix of the key i itself; If there exists i < j such that M(F(key i-1 )) < M(F(key i )) = M(F(key i+1 )) = … = M(F(key j )) < M(F(key j+1 )),then recursively call the new node construction operation with {key i ,…,key j} as the input, and determine the entry with the serial number M(F(key i )) as the node entry, where the node entry is used to store a memory address pointer pointing to the new node; The sequence number is not equal to any M(F(key i )) is determined as the empty entry.

6. The method for operating a learning index tree graph according to claim 5, characterized in that: The new node construction operation includes: The new node is recorded as node, the input key required to construct node is obtained, and the common prefix of node is set to the longest common prefix of all the input keys; Determine the number of entries of the node according to the number of input keys, the lower limit min of the number of entries, and the upper limit max of the number of entries, wherein the method for determining the number of entries of the node is consistent with the method for determining the number of entries of the root node; All the input keys are trimmed of the common prefix of node, and the common prefix is ​​substituted into the conversion function F for calculation to determine the linear model of node, and the method for determining the linear model of node is consistent with the method for determining the linear model of the root node; Determine multiple entries of the node according to the input key and the linear model of the node, wherein the method for determining the entry type of the entry in the node is consistent with the method for determining the entry type of the entry in the root node; When a certain entry of node is determined as the data entry, a byte array obtained by trimming the common prefix of node from the corresponding input key is determined as the suffix of the data entry; When a certain entry of node is determined to be the node entry, the new node construction operation is recursively called again, and any input key during the call is a byte array obtained by trimming off the common prefix of node.

7. The method for operating a learning index tree graph according to claim 1, characterized in that: The point query includes: Determine the target key to be queried in the instruction content of the point query, return locate(the root node, the target key), the first parameter node of the locate(node, key) function is the node in the learning index tree graph, the second parameter key is the byte array to be queried, the locate(node, key) function substitutes key into the linear model M of node for processing, and obtains the target node sequence number M(F(key)); determine the entry corresponding to the target node sequence number M(F(key)), which is called the query entry; Determining the item type of the query item; When the item type of the query item is an empty item, marking the query item as a positioning item of the point query and returning information that the target key cannot be found; When the entry type of the query entry is a data entry, the query entry is marked as a positioning entry of the point query, and the common prefix of all nodes in the path from the root node to the node where the query entry is located is obtained, which is called the search prefix, and the suffix stored in the query entry, which is called the search suffix, and the result of splicing all the search prefixes and the search suffixes is marked as a comparison key; if the comparison key is not equal to the target key, the information that the target key cannot be found is returned; if the comparison key is equal to the target key, the corresponding value pointed to by the disk address pointer of the value log area stored in the query entry is returned; When the entry type of the query entry is a node entry, the child node is accessed through the memory address pointer of the lower-level child node stored in the query entry. If the common prefix of the child node is also the prefix of the key, the common prefix of the child node is cut off from the key, and the remaining part of the key is recorded as the key. rem , and returns locate(the child node, key rem ), otherwise the query entry is marked as a positioning entry of the point query and the information that the target key cannot be found is returned; The range query includes: Determine a range lower limit query key and a range upper limit query key to be queried in the range query instruction content; Execute the point query according to the range lower limit query key, determine the location entry of the point query, and mark it as the start entry; Execute the point query according to the range upper limit query key, determine the location entry of the point query, and mark it as the end entry; Traverse the data entries in all nodes between the start entry and the end entry in a depth-first manner; for each of the data entries, obtain the common prefix of all nodes in the path from the root node to the node where the data entry is located, which is called the search prefix, and the suffix stored in the data entry, which is called the search suffix, and mark the result of concatenating all the search prefixes and the search suffixes as the corresponding key of the data entry; return the corresponding keys of all the data entries and the corresponding values ​​pointed to by the disk address pointer of the value log area stored therein.

8. The method for operating a learning index tree graph according to claim 1, characterized in that: The adding operation includes: Determine the disk address pointer of the new key to be added in the instruction content of the add operation and the value of the new key in the value log area, and record the new key as key new , key new The disk address pointer of the value in the value log area is recorded as point new ; for key new Execute the point query to determine a location entry for the point query, referred to as a first target entry; Determine each node in the path from the root node to the first target entry, mark it as a path node; concatenate the common prefixes of all the path nodes and mark the result as prefix; new The result after trimming off the prefix is ​​marked as suffix new ; determining the entry type of the first target entry; If the first target entry is an empty entry, the entry type of the first target entry is changed to a data entry, and suffix is ​​stored in the first target entry. new and point new ; If the first target entry is a data entry, the original suffix of the first target entry is obtained, which is recorded as suffix old ; if suffix old with suffix new If the first target entry is consistent, the log area disk address pointer is changed to point new , otherwise {suffix new ,suffix old } is used as input to call the new node construction operation to construct the first new node, which is the subtree with suffix in the root. new The corresponding data entry stores suffix new and point new , and changing the first target entry to a node entry for storing a memory address pointer pointing to the first new node; If the first target entry is a node entry, the child node pointed to by the first target entry is marked as an invalid node; all remaining keys and suffixes covered in the subtree with the invalid node as the root are new The new node construction operation is called as input to construct a second new node, the second new node is the subtree with suffix in the root. new The corresponding data entry stores suffix new and point new , each of the remaining keys is formed by concatenating the common prefix of all nodes in the path from the invalid node to a data entry and the suffix of the data entry; replacing the child node pointer stored in the first target entry with the pointer of the second new node, and deleting the invalid node; Determine whether each of the path nodes meets a preset reconstruction condition, where the reconstruction condition is that the total number of nodes in the subtree where the path node is located is not less than a preset node number threshold, and is not less than a preset multiple of the number of entries in the entry list of the path node, and the proportion of data entries in all entries covered by the subtree where the path node is located is not less than a preset proportion; When the path node meets the reconstruction condition, the path node is marked as a reconstruction node; a new node construction operation is called with the set of all remaining keys covered in the subtree with the reconstruction node as the root as input to construct a third new node, each of the remaining keys is composed of the common prefix of all nodes in the path from the reconstruction node to a data entry and the suffix of the data entry; the child node pointer stored in the node entry pointing to the reconstruction node is replaced with the pointer to the third new node, and the reconstruction node is deleted.

9. The method for operating a learning index tree graph according to claim 1, wherein: The deletion operation includes: Determine the delete key to be deleted in the instruction content of the delete operation, and record the delete key as key old , for key old Execute the point query to determine a location entry for the point query, referred to as a second target entry; If the point query returns key old If no information is found, the deletion operation is terminated; otherwise, the entry type of the second target entry is changed to an empty entry.

10. The method for operating a learning index tree graph according to claim 1, wherein: The modification operation includes: Determine the disk address pointer of the original key before modification, the new key after modification and the value of the new key in the value log area in the instruction content of the modification operation, and record the original key before modification as key old , the modified new key is recorded as key alter , key alter The disk address pointer of the value in the value log area is recorded as point alter ; If key old With key alter If they are consistent, then the key alter Execute the point query to determine the location entry of the point query, which is called the third target entry; if the point query returns key alter If the information is not found, the key alter and point alter As the instruction content, the add operation is executed; otherwise, the value log area disk address pointer stored in the third target entry is changed to point alter ; If key old With key alter If they are not the same, first use key alter and point alter As the instruction content, execute the adding operation, and then use key old The deletion operation is performed as the instruction content.

Images