A storage engine caching method and system based on ART tree

Abstract

The present invention discloses a storage engine cache method and system based on ART tree, which belongs to the field of database. By means of data index migration between two ART trees, a new ART tree is established and the old ART tree is eliminated; a threshold of data index migration is set, and when the percentage of the memory space occupied by the currently cached KV data to the set maximum cache value is greater than or equal to the threshold of the data index migration, the data index migration work is started; when the memory space of the KV data of the new and old ART tree indexes is equal to or exceeds the set maximum cache value, the old ART tree is deleted, and the KV data storage space of the old ART tree index is released. The present invention can optimize memory usage and maintain the query performance of the ART tree.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a storage engine cache method and system based on an ART tree. Background Art

[0002] The way data is organized in memory has a direct impact on the throughput of the storage engine. The ART algorithm is an index algorithm that has just appeared in the past two years. Among many index algorithms, ART currently has the best insertion and query performance and is faster than the hash algorithm.

[0003] ART stands for Adaptive Radix Tree, which is actually an upgraded version of Radix Tree. Radix Trie (also called Radix Trie or Compressed Prefix Tree) is a data structure, a more space-saving Trie (prefix tree), in which each node as the only child node is merged with its parent node, and the edge can be represented as a sequence of elements or as a single element.

[0004] The ART algorithm greatly improves query performance by using path compression and lazy expansion technology. However, when the ART algorithm deletes some data, if the path is not recompressed, the access performance of the entire ART tree will be reduced; if the path is recompressed, the implementation logic is very complex and has a great impact on the performance of the ART algorithm at runtime. Summary of the invention

[0005] The technical task of the present invention is to address the above deficiencies and provide a storage engine caching method and system based on an ART tree, which can optimize memory usage and maintain the query performance of the ART tree.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A storage engine cache method based on an ART tree, which establishes a new ART tree and eliminates an old ART tree by migrating data indexes between two ART trees;

[0008] Set a threshold for data index migration, and start data index migration when the percentage of memory space occupied by the currently cached KV data to the set maximum cache value is greater than or equal to the threshold for data index migration;

[0009] When the memory space of the KV data indexed by the old and new ART trees is equal to or exceeds the set maximum cache value, the old ART tree is deleted and the KV data storage space of the old ART tree index is released.

[0010] The operation of the storage engine program is limited by the size of the allocated memory space, and it is impossible to cache all the data in the memory. Therefore, it is necessary to eliminate the infrequently used data from the memory.

[0011] This method eliminates the old ART tree by migrating the data index between two ART trees, and the remaining new ART tree has better access performance. At the same time, in the process of data index migration, the memory data that is not frequently used is eliminated.

[0012] Aiming at the phenomenon that the memory space occupied by the storage engine cache is limited and the performance of the ART algorithm decreases as the amount of indexed data increases, the storage engine cache mechanism based on the ART tree provided by this method can optimize memory usage and maintain the query performance of the ART tree.

[0013] Furthermore, the implementation process of this method includes executing the logic of a single ART tree query and executing the logic of a double ART tree query. When starting to query a key value, first determine whether there is only one ART tree in the current cache. If there is only one, then follow the logic of a single ART tree query; if there are two, then follow the logic of a double ART tree query; and finally return the queried KV data.

[0014] Preferably, a threshold for preferentially querying the new ART tree is set, and the threshold for preferentially querying the new ART tree is the ratio of the amount of data in the new ART tree to the amount of data in the original ART tree. If it is greater than the threshold, the new ART tree is searched first; if it is less than the threshold, the old ART tree is searched first.

[0015] Furthermore, the logic of the single ART tree query is as follows:

[0016] First, use the query algorithm in the ART algorithm to query whether the Key value exists in the ART tree. If it is found, the corresponding KV data is directly returned; if it is not found, the corresponding KV data needs to be extracted from the disk or other media, and the KV data is indexed on the ART tree;

[0017] Mounting new KV data will occupy new memory. Therefore, it is necessary to determine whether the memory space occupied by the current KV data exceeds the threshold of data index migration. If it does not exceed, the KV data is returned directly; if it exceeds, a new ART tree is created, a dual ART tree structure is formed in the cache, and then the KV data is returned.

[0018] Furthermore, the logic of the dual ART tree query is as follows:

[0019] S1: Determine whether to query the original ART tree first, according to the judgment rule whether the ratio of the data volume in the new ART tree to the data volume in the original ART tree is greater than the threshold of the new ART tree being queried first. If it is greater than, it means to query the new ART tree first and proceed to the next step; if it is less than or equal to the threshold of the new ART tree being queried first, it means to query the original ART tree first and jump to step S4;

[0020] S2: Use the query algorithm in the ART algorithm to query whether the Key value exists in the new ART tree. If it is found, jump to step S9; if it is not found, go to the next step;

[0021] S3: Use the query algorithm in the ART algorithm to query whether the Key value exists in the original ART tree. If found, index the KV data on the new ART tree, delete the index of the KV data on the original ART tree, and jump to step S9; if not found, read the KV data corresponding to the Key value from the disk, index the KV data on the new ART tree, and jump to step S6;.

[0022] S4: Use the query algorithm in the ART algorithm to query whether the Key value exists in the original ART tree. If it is found, create an index for the KV data on the new ART tree, delete the index of the KV data on the original ART tree, and jump to step S9. If it is not found, go to the next step.

[0023] S5: Use the query algorithm in the ART algorithm to query whether the Key value exists in the new ART tree. If it is found, jump to step S9; if it is not found, read the KV data corresponding to the Key value from the disk, and create an index for the KV data on the new ART tree, and proceed to the next step;

[0024] S6: Determine whether the memory space occupied by the current KV data is equal to or exceeds the set maximum cache value. If not, jump to step S9; if yes, it means that the cache space is full and data needs to be eliminated, and go to the next step;

[0025] S7: Delete the original ART tree and release the memory space occupied by the KV data indexed by the original ART tree; (Delete the old ART tree. The most recently accessed KV data in the old ART tree has been migrated to the new ART tree. Therefore, the remaining KV data in the old ART tree can be eliminated. Therefore, directly delete the old ART tree to release the indexed KV data space) At this time, the storage engine has only one ART tree; go to the next step;

[0026] S8: Based on the statistical query information, recalculate the threshold for data index migration and the threshold for prioritizing querying the new ART tree, and proceed to step S10;

[0027] S9: Count the query information and proceed to the next step.

[0028] S10: Return the corresponding KV data and end.

[0029] Preferably, the threshold of data index migration is set to P, and the calculation method of the threshold P is as follows:

[0030] When deleting the original ART tree, the amount of data deleted is set to D; then the calculation formula for the threshold P is:

[0031]

[0032] Where Pi represents the threshold calculated this time, Di represents the amount of data deleted this time, S represents the maximum cache space; e is a mathematical constant, which is the base of the natural logarithm function.

[0033] Preferably, the threshold for preferentially searching the new ART tree is set to SEEK_NEW_FIRST_TH, and the calculation method of the threshold SEEK_NEW_FIRST_TH is as follows:

[0034] In the statistical query information, when the number of hits of the new ART tree M 1 Equal to the number of hits in the original ART tree M 2 When , the ratio of the amount of data in the new ART tree to the amount of data in the original ART tree is set to M;

[0035] The calculation formula of the threshold SEEK_NEW_FIRST_TH is:

[0036]

[0037] The present invention also claims protection for a storage engine cache system based on an ART tree, which implements KV data storage through the above-mentioned storage engine cache method based on an ART tree.

[0038] The present invention also claims protection for a storage engine cache device based on an ART tree, comprising: at least one memory and at least one processor;

[0039] The at least one memory is used to store a machine-readable program;

[0040] The at least one processor is used to call the machine-readable program to execute the above-mentioned storage engine caching method based on the ART tree.

[0041] The present invention also requires protection of a computer-readable medium having computer instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned storage engine caching method based on the ART tree.

[0042] Compared with the prior art, the storage engine cache method and system based on ART tree of the present invention has the following beneficial effects:

[0043] 1. This method can be used to implement range query function in cache;

[0044] 2. Cache query performance is improved by about 10% compared with hash query;

[0045] 3. The complexity of KV data elimination is lower than that of LRU and other algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a structural diagram of a storage engine KV data cache based on an ART tree provided in an embodiment of the present invention;

[0047] Figure 2 It is an overall flow chart of using cache by a storage engine based on an ART tree provided by an embodiment of the present invention;

[0048] Figure 3 is a query flow diagram of a single ART tree provided by an embodiment of the present invention;

[0049] Figure 4 is a query flow diagram of a dual ART tree provided in an embodiment of the present invention;

[0050] Figure 5 is a diagram of cached KV data for index migration provided by an embodiment of the present invention;

[0051] Figure 6 It is a diagram illustrating caching new KV data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0053] The embodiment of the present invention proposes a storage engine cache method based on an ART tree, which establishes a new ART tree and eliminates the old ART tree by migrating data indexes between two ART trees.

[0054] Set a threshold for data index migration, and start data index migration when the percentage of memory space occupied by the currently cached KV data to the set maximum cache value is greater than or equal to the threshold for data index migration;

[0055] When the memory space of the KV data indexed by the old and new ART trees is equal to or exceeds the set maximum cache value, the old ART tree is deleted and the KV data storage space of the old ART tree index is released.

[0056] The operation of the storage engine program is limited by the size of the allocated memory space, and it is impossible to cache all the data in the memory. Therefore, it is necessary to eliminate the infrequently used data from the memory.

[0057] This method eliminates the old ART tree by migrating the data index between two ART trees, and the remaining new ART tree has better access performance. At the same time, in the process of data index migration, the memory data that is not frequently used is eliminated.

[0058] Aiming at the phenomenon that the memory space occupied by the storage engine cache is limited and the performance of the ART algorithm decreases as the amount of indexed data increases, the storage engine cache mechanism based on the ART tree provided by this method can optimize memory usage and maintain the query performance of the ART tree.

[0059] like Figure 1 The figure shows the structure of the storage engine KV data cache in this method. The triangle represents the ART tree structure, and each box inside represents a node in the ART tree. The ART tree child nodes point to the KV data in the memory. At the same time, the KV data in the memory is sorted according to the size of the key to form a linked list. The role of the linked list is to support the range query function.

[0060] The overall process of this method is as follows:

[0061] When the storage engine is started, the maximum cache size is set to S, which will not change during this run;

[0062] When the storage engine starts, there is only one ART tree and no cached KV data;

[0063] The storage engine has several variables:

[0064] (1) The threshold value P of data index migration, where P is the percentage of the memory space occupied by the currently cached KV data in S. When the percentage is greater than or equal to P, the data index migration work is started. The threshold value P is initially set to 0.7.

[0065] (2) The threshold SEEK_NEW_FIRST_TH for preferentially querying the new ART tree is expressed as the ratio of the amount of data in the new ART tree to the amount of data in the original ART tree. The threshold SEEK_NEW_FIRST_TH is initially set to 0.7.

[0066] The overall process of using cache by storage engine is as follows Figure 2 As shown:

[0067] When starting to query a key value, first determine whether there is only one ART tree in the current cache. If there is only one, use the logic of single ART tree query; if there are two, use the logic of dual ART tree query. Finally, return the queried KV data.

[0068] Single ART tree query logic:

[0069] The general query process of a single ART tree is as follows Figure 3 As shown, first use the query algorithm in the ART algorithm to query whether the Key value exists in the ART tree. If it is found, the corresponding KV data is directly returned; if it is not found, it is necessary to extract the corresponding KV data from the disk or other media and index the KV data on the ART tree;

[0070] Mounting new KV data will occupy new memory. Therefore, it is necessary to determine whether the memory space occupied by the current KV data exceeds the threshold P. If not, the KV data is returned directly. If it exceeds the threshold P, a new ART tree is created, a dual ART tree structure is formed in the cache, and then the KV data is returned.

[0071] Dual ART tree query logic:

[0072] The general query process of the double ART tree is as follows Figure 4 shown.

[0073] Step 1: Determine whether to query the original ART tree first. The judgment rule is whether the ratio of the data volume in the new ART tree to the data volume in the original ART tree is greater than the threshold SEEK_NEW_FIRST_TH. If it is greater, it means to query the new ART tree first and proceed to the next step; if it is less than or equal to the threshold SEEK_NEW_FIRST_TH, it means to query the original ART tree first and jump to the fourth step.

[0074] Step 2: Use the query algorithm in the ART algorithm to query whether the Key value exists in the new ART tree. If it is found, jump to step 9; if not, go to the next step.

[0075] Step 3: Use the query algorithm in the ART algorithm to query the original ART tree for the existence of the Key value. If found, create an index for the KV data on the new ART tree and delete the index of the KV data on the original ART tree (such as Figure 5 As shown: Index migration cached KV data), jump to step 9; if no query is found, read the KV data corresponding to the Key value from the disk and create an index on the new ART tree (as shown Figure 6 As shown: cache new KV data), jump to step 6.

[0076] Step 4: Use the query algorithm in the ART algorithm to query whether the Key value exists in the original ART tree. If it is found, create an index for the KV data on the new ART tree, delete the index of the KV data on the original ART tree, and jump to step 9; if it is not found, go to the next step.

[0077] Step 5: Use the query algorithm in the ART algorithm to query whether the Key value exists in the new ART tree. If it is found, jump to step 9; if it is not found, read the KV data corresponding to the Key value from the disk, and index the KV data on the new ART tree, and proceed to the next step.

[0078] Step 6: Determine whether the memory space occupied by the current KV data is equal to or exceeds the threshold S. If not, jump to step 9; if yes, it means that the cache space is full and data needs to be eliminated, and then go to the next step.

[0079] Step 7: Delete the original ART tree, the KV data indexed by the original ART tree, and the memory space they occupy. At this point, the storage engine has only one ART tree. Go to the next step.

[0080] Step 8: Based on the statistical query information, recalculate the threshold P and the threshold SEEK_NEW_FIRST_TH. Go to step 10.

[0081] Step 9: Count the query information and proceed to the next step.

[0082] Step 10: Return the corresponding KV data. End.

[0083] The threshold is calculated as follows:

[0084] 1. The threshold for data index migration is set to P. The calculation method of the threshold P is as follows:

[0085] When deleting the original ART tree, the amount of data deleted is set to D; then the calculation formula for the threshold P is:

[0086]

[0087] Where Pi represents the threshold calculated this time, Di represents the amount of data deleted this time, S represents the maximum cache space; e is a mathematical constant, which is the base of the natural logarithm function.

[0088] 2. The threshold for preferentially searching the new ART tree is set to SEEK_NEW_FIRST_TH. The calculation method of the threshold SEEK_NEW_FIRST_TH is as follows:

[0089] In the statistical query information, when the number of hits of the new ART tree M 1 Equal to the number of hits in the original ART tree M 2When , the ratio of the amount of data in the new ART tree to the amount of data in the original ART tree is set to M;

[0090] The calculation formula of the threshold SEEK_NEW_FIRST_TH is:

[0091]

[0092] The embodiment of the present invention further provides a storage engine cache system based on an ART tree, which implements KV data storage through the storage engine cache method based on an ART tree described in the above embodiment of the present invention.

[0093] The embodiment of the present invention also provides a storage engine cache device based on an ART tree, comprising: at least one memory and at least one processor;

[0094] The at least one memory is used to store a machine-readable program;

[0095] The at least one processor is used to call the machine-readable program to execute the storage engine cache method based on the ART tree described in the above embodiment of the present invention.

[0096] The embodiment of the present invention further provides a computer-readable medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the processor executes the storage engine cache method based on the ART tree described in the above embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0097] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0098] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0099] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0100] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0101] The present invention is shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A storage engine caching method based on ART tree, characterized in that: By migrating data indexes between two ART trees, a new ART tree is established and the old ART tree is eliminated. Set a threshold for data index migration, and start data index migration when the percentage of memory space occupied by the currently cached KV data to the set maximum cache value is greater than or equal to the threshold for data index migration; When the memory space of the KV data indexed by the old and new ART trees is equal to or exceeds the set maximum cache value, the old ART tree is deleted and the KV data storage space of the old ART tree index is released; Set the threshold for first querying the new ART tree. The threshold for first querying the new ART tree is the ratio of the amount of data in the new ART tree to the amount of data in the original ART tree. If it is greater than the threshold, the new ART tree is searched first. If it is less than the threshold, the old ART tree is checked first.

2. A storage engine caching method based on ART tree according to claim 1, characterized in that The implementation process of this method includes executing the logic of a single ART tree query and executing the logic of a double ART tree query. When starting to query a key value, first determine whether there is only one ART tree in the current cache. If there is only one, the logic of a single ART tree query is used; if there are two, the logic of a double ART tree query is used; and finally the queried KV data is returned.

3. A storage engine caching method based on ART tree according to claim 2, characterized in that: The logic of the single ART tree query is as follows: First, use the query algorithm in the ART algorithm to query whether the Key value exists in the ART tree. If it is found, the corresponding KV data is directly returned; if it is not found, the corresponding KV data needs to be extracted from the disk or other media, and the KV data is indexed on the ART tree; Mounting new KV data will occupy new memory. Therefore, it is necessary to determine whether the memory space occupied by the current KV data exceeds the threshold for data index migration. If not, the KV data is returned directly. If it exceeds, a new ART tree is created, a dual ART tree structure is formed in the cache, and then the KV data is returned.

4. The storage engine caching method based on ART tree according to claim 3, characterized in that: The logic of double ART tree query is as follows: S1: Determine whether to query the original ART tree first, according to the judgment rule whether the ratio of the data volume in the new ART tree to the data volume in the original ART tree is greater than the threshold for preferentially querying the new ART tree. If it is greater, it means that the new ART tree is searched first and the next step is entered; If it is less than or equal to the threshold for querying the new ART tree first, it means that the original ART tree is searched first, and the process goes to step S4; S2: Use the query algorithm in the ART algorithm to query whether the Key value exists in the new ART tree. If it is found, jump to step S9; if it is not found, go to the next step; S3: Use the query algorithm in the ART algorithm to query whether the Key value exists in the original ART tree. If it is found, the KV data is indexed on the new ART tree, and the index of the KV data on the original ART tree is deleted, and the process goes to step S9; if it is not found, read the KV data corresponding to the Key value from the disk, and index the KV data on the new ART tree, and the process goes to step S6; S4: Use the query algorithm in the ART algorithm to query whether the Key value exists in the original ART tree. If it is found, create an index for the KV data on the new ART tree, delete the index of the KV data on the original ART tree, and jump to step S9. If it is not found, go to the next step. S5: Use the query algorithm in the ART algorithm to query whether the Key value exists in the new ART tree. If it is found, jump to step S9; if it is not found, read the KV data corresponding to the Key value from the disk, and create an index for the KV data on the new ART tree, and proceed to the next step; S6: Determine whether the memory space occupied by the current KV data is equal to or exceeds the set maximum cache value; If not, jump to step S9; If yes, it means the cache space is full and data needs to be eliminated, and then go to the next step; S7: Delete the original ART tree and release the memory space occupied by the KV data indexed by the original ART tree; At this point, the storage engine has only one ART tree; Go to the next step; S8: Based on the statistical query information, recalculate the threshold for data index migration and the threshold for prioritizing querying the new ART tree, and proceed to step S10; S9: collect query information and proceed to the next step; S10: Return the corresponding KV data and end.

5. A storage engine caching method based on an ART tree according to any one of claims 1 to 4, characterized in that: The threshold for data index migration is set to P. The calculation method of threshold P is as follows: When deleting the original ART tree, the amount of data deleted is set to D; then the calculation formula for the threshold P is: ； Where P i Indicates the threshold calculated this time, D i Indicates the amount of data deleted this time, S indicates the maximum cache space; e is a mathematical constant, which is the base of the natural logarithm function.

6. The storage engine caching method based on ART tree according to claim 5, characterized in that: The threshold for first searching the new ART tree is set to SEEK_NEW_FIRST_TH. The calculation method of the threshold SEEK_NEW_FIRST_TH is as follows: In the statistical query information, when the number of hits in the new ART tree M1 is equal to the number of hits in the original ART tree M2, the ratio of the amount of data in the new ART tree to the amount of data in the original ART tree is set to M i ; The calculation formula of the threshold SEEK_NEW_FIRST_TH is: ； In the formula, S i-1 is the threshold SEEK_NEW_FIRST_TH calculated last time, S i This is the threshold SEEK_NEW_FIRST_TH calculated this time.

7. A storage engine cache system based on ART tree, characterized in that: The system implements KV data storage through the storage engine caching method based on the ART tree described in any one of claims 1-6.

8. A storage engine cache device based on ART tree, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 6.

9. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, and when the computer instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 6.

Images