A query method and system for proxy storage engine under storage and computing separation architecture

Abstract

The present invention discloses a query method and system for a proxy storage engine under a storage-computation separation architecture, which belongs to the field of database technology and includes a computing node and a data node. The computing node checks whether required data is cached locally through a proxy storage engine. If all data are in the cache and the storage engine unique identifiers of the data are consistent, the cached data is used for query; if only part of the data is in the cache, the required data and the corresponding new storage engine unique identifier are obtained from the data node. The data node is responsible for calculating the new storage engine unique identifier corresponding to the stored data, and comparing the new storage engine unique identifier with the storage engine unique identifier of the local corresponding cached data. If they are consistent, the required data is used for query, and the missing data and the corresponding storage engine unique identifier are cached locally; if they are inconsistent, the required data is obtained from the data node for query, and the obtained data and its corresponding new storage engine unique identifier are cached locally.

Description

Technical Field

[0001] The present invention belongs to the field of database technology, and specifically relates to a query method and system for a proxy storage engine under a storage-computation separation architecture. Background Art

[0002] The DolphinDB database system supports multi-modal storage engines, and each storage engine has specific advantages for different query loads. However, if a unified data cache method is used for each storage engine in the computing node, the computing node will not be able to fully utilize the advantages of each storage engine under different query loads.

[0003] The primary key engine PKEY is a storage engine implemented at the bottom of the database system DolphinDB based on the LSM-Tree (LogStructured MergeTree) architecture. While maintaining the uniqueness of the primary key, it provides excellent primary key column query performance. At the same time, the primary key engine PKEY can create sparse indexes for primary keys and non-primary key columns, and the query efficiency of sparse indexes of non-primary key columns is very high. Sparse index is a common index structure in databases that can speed up retrieval during queries. Its main feature is that it does not create indexes for every data item, but only creates indexes for some data items. However, when the storage structure of the primary key engine PKEY changes, there is a problem of inconsistent data arrangement between the computing node cache and the data node, which affects the accuracy of the query results. Summary of the invention

[0004] In order to solve the shortcomings of the prior art and achieve the purpose of fully utilizing the query performance of the storage engine through the proxy engine in the storage-computation separation architecture, the present invention adopts the following technical solutions:

[0005] A query method for a proxy storage engine in a storage-computation separation architecture, applied to a computing node, comprises the following steps:

[0006] Step 1: Use the PKEY proxy storage engine to check whether all required column data is cached locally. If all column data is in the cache and the storage engine unique identifier of the column data is consistent (the primary key engine PKEY version ID is consistent), execute step 2; if only part of the column data is in the cache, execute step 3;

[0007] Step 2: Use cached data directly for query;

[0008] Step 3: Obtain the required column data and the new storage engine unique identifier calculated based on the required column data from the data node, and compare the new storage engine unique identifier with the storage engine unique identifier of the corresponding locally cached column data. If they are consistent, execute step 4; if they are inconsistent, execute step 5;

[0009] Step 4: Use the required column data to query, and cache the missing column data and the corresponding storage engine unique identifier locally;

[0010] Step 5: Mark the cached data as invalid, re-obtain the required column data from the data node for query, and cache the obtained column data and its corresponding new storage engine unique identifier locally.

[0011] Furthermore, the required data obtained from the data node is the column data stored in column form by the storage engine of the data node. The obtained column data includes a set of column data segments, and the column data segments are equipped with sparse indexes. When querying, the PKEY proxy storage engine quickly filters the column data segments through the sparse indexes, thereby improving query efficiency.

[0012] Furthermore, the obtained column data is stored in columnar form through the primary key engine on the data node. The sparse index of its column data segment is a Bloom filter. Each column data segment of the primary key column contains a Bloom filter. The PKEY proxy storage engine performs primary key point query based on the Bloom filter, which can be extremely efficient.

[0013] Furthermore, obtaining the storage engine unique identifier from the data node is a version ID calculated based on metadata corresponding to the required data stored in the storage engine metadata version of the data node.

[0014] A query method for a proxy storage engine in a storage-computation separation architecture, applied to a data node;

[0015] Calculate the storage engine unique identifier corresponding to the stored column data so that the computing node caches the column data and its corresponding storage engine unique identifier, and when the PKEY proxy storage engine queries, check whether the required column data is in the cache and the storage engine unique identifier of the column data is consistent (the primary key engine PKEY version ID is consistent). If so, directly use the cached data for query. If only part of the column data is in the cache, calculate the new storage engine unique identifier based on the column data required by the computing node to compare it with the storage engine unique identifier cached locally by the computing node. If they are consistent, enable the computing node to use the required column data for query, and cache the missing column data and the corresponding storage engine unique identifier locally in the computing node. If they are inconsistent, send the required column data to the computing node for query, and enable the computing node to cache the acquired column data and its corresponding new storage engine unique identifier locally.

[0016] Furthermore, the storage engine stores data in columns at the data nodes. In order to give full play to the characteristics of column storage, the primary key engine PKEY caches the computing nodes at a fine-grained level based on the entire column of data. The column cache of the storage engine includes a set of column data segments, each of which represents a storage unit of the storage engine. Each column data segment is equipped with a sparse index (such as ZoneMap, Bloom filter) so that when the computing node is queried, the PKEY proxy storage engine can quickly filter the column data segments through the sparse index, thereby improving query efficiency.

[0017] Further, the storage unit includes a data storage unit Level File where PKEY is located in a disk and a data storage unit MemTable where PKEY is located in a memory;

[0018] The PKEY is located in the column data segment corresponding to the data storage unit Level File in the disk, and is arranged in the order of new data first and old data last. For the column data segment of the data storage unit Level File in the disk, since the sparse index of these data has been stored in the data storage unit Level File in the disk, it can be directly reused without regeneration;

[0019] The PKEY is located in the data storage unit MemTable in the memory. If the data storage unit MemTable exists in the memory, the corresponding column data segment is arranged at the end.

[0020] Furthermore, the storage engine is a primary key engine, and the sparse index of its column data segment is a Bloom filter. Each column data segment of the primary key column contains a Bloom filter, so that the PKEY proxy storage engine can perform primary key point query based on the Bloom filter, which can be extremely efficient.

[0021] Furthermore, the data node calculates the storage engine PKEY version ID based on the metadata of the storage unit in its current storage engine metadata version PKVersion, and sends it to the computing node together with the column data to compare it with the storage engine PKEY version ID in the local cache of the computing node.

[0022] A query system for a proxy storage engine in a storage-computation separation architecture, comprising a computing node and a data node;

[0023] The computing node uses the PKEY proxy storage engine to check whether all required column data are cached locally. If all column data are in the cache and the storage engine unique identifier of the column data is consistent (the primary key engine PKEY version ID is consistent), the cached data is directly used for query; if only part of the column data is in the cache, the required column data and the corresponding new storage engine unique identifier are obtained from the data node, and the new storage engine unique identifier is compared with the storage engine unique identifier of the corresponding locally cached column data. If they are consistent, the required column data is used for query, and the missing column data and the corresponding storage engine unique identifier are cached locally; if they are inconsistent, the cached data is marked as invalid, the required column data is obtained from the data node again for query, and the obtained column data and its corresponding new storage engine unique identifier are cached locally;

[0024] The data node calculates the new storage engine unique identifier corresponding to the stored column data, and sends it to the computing node for comparison. When the comparison is consistent, the required column data is sent to the computing node.

[0025] The advantages and beneficial effects of the present invention are:

[0026] The present invention designs and implements a PKEY proxy engine based on the characteristics of a PKEY storage engine, which can cache column data and corresponding sparse indexes on data nodes on computing nodes, so that computing nodes can cache data containing indexes that have been built by data nodes, thereby enabling the PKEY proxy engine to give full play to the query performance advantages of the PKEY storage engine on computing nodes. The present invention also generates a unique version ID for PKEY metadata using a hash algorithm, thereby solving the problem of inconsistent arrangement order between column and column data of different versions. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the format of the PKEY proxy engine data column cache in an embodiment of the present invention.

[0028] Figure 2 It is a flow chart of a computing node obtaining data through a PKEY proxy engine in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0030] A query system for a proxy storage engine in a storage-computation separation architecture includes a computing node and a data node.

[0031] like Figure 1As shown in the figure, the primary key engine PKEY is stored in a column-based manner in the data node. In order to give full play to the characteristics of column-based storage, the cache of the primary key engine PKEY in the computing node is performed at a fine-grained level based on the entire column of data. The column cache of the primary key engine PKEY consists of multiple column data segments, each of which represents a storage unit (Level File or MemTable) of the primary key engine PKEY. For the storage unit Level File (the data storage unit where PKEY is located on the disk, which stores the data storage file), the corresponding column data segments are arranged in the order of new and old, with new data in the front and old data in the back; for the storage unit MemTable, if the storage unit MemTable (the data storage unit where PKEY is located in the memory) has data, its corresponding column data segment is arranged at the end. Each column data segment is equipped with a sparse index (such as ZoneMap, Bloom filter). Among them, for the column data segments from the storage unit Level File, since the sparse indexes of these data are already stored in the storage unit Level File, they can be directly reused without regenerating. When querying, the PKEY proxy engine quickly filters the column data segments through these sparse indexes, thereby improving query efficiency. At the same time, each column data segment of the primary key column contains a Bloom filter, which makes the PKEY proxy engine extremely efficient when querying the primary key. The proxy engine is a computing node cache layer implemented for different storage engines under the storage and computing separation architecture of the DolphinDB database system, which is used to obtain query data.

[0032] When the primary key engine PKEY performs disk flushing or data merging operations, the storage unit Level File in the disk may be added or deleted, and the storage unit Level File metadata stored in the primary key engine metadata version PKVersion (a data structure used to manage PKEY metadata) will also change accordingly. Even if no new data is written, the column data of the same table may have inconsistent column data arrangement order due to the background disk flushing or data merging operations of the primary key engine PKEY.

[0033] In order to solve this problem, the present invention generates a primary key engine PKEY version ID for each primary key engine metadata version PKVersion. Specifically, when the data node provides the required data to the computing node, the data node generates the primary key engine PKEY version ID through a hash encryption algorithm based on the storage unit Level File metadata in the current primary key engine metadata version PKVersion, and sends the ID together with the column data to the computing node.

[0034] For example, the compute node requests the data node to obtain the data of column Col0. The data node will generate the primary key engine PKEY version ID as a unique identifier based on the current primary key engine metadata version PKVersion. Assume that the primary key engine metadata version PKVersion stores the metadata information of three LevelFiles, and the names of these three files are 0-000-0, 0-000-1, and 1-001-1. The data node first calculates the primary key engine PKEY version ID by using a hash encryption algorithm (such as SHA1) to calculate the file names of these three storage unit LevelFiles. Taking the SHA1 algorithm as an example, the generated primary key engine PKEY version ID is 06759556e48d4f743dd61d13f9b660d8b20967b4. Subsequently, the data node extracts the data of column Col0 and the corresponding sparse index from these three LevelFiles. The column data and index of each storage unit LevelFile form a column data segment, so a total of three column data segments are generated. Finally, the data node sends these three column data segments together with the generated primary key engine PKEY version ID to the compute node.

[0035] like Figure 2 As shown, a query method of a proxy storage engine in a storage-computation separation architecture, where a computing node obtains table data by calling a PKEY proxy storage engine, includes the following steps:

[0036] Step 1: Use the PKEY proxy storage engine to check whether the current computing node has cached all the required column data. If all the column data is in the cache and the primary key engine PKEY version IDs of these columns are consistent, execute step 2; if only some of the column data is in the computing node cache, execute step 3;

[0037] Step 2: Use cached data directly for query;

[0038] Step 3: Get the data of these columns from the data node. The data node will calculate the primary key engine PKEY version ID based on its current primary key engine metadata version PKVersion and send it to the computing node together with the column data. After receiving the data from the data node, the computing node compares the primary key engine PKEY version ID in the local cache with the version ID of the data obtained from the data node. If the version ID comparison results are consistent, execute step 4; if the version ID comparison results are inconsistent, execute step 5;

[0039] Step 4: Use this data to query and cache the missing columns on the compute nodes;

[0040] Step 5: Mark the cached data as invalid, obtain the required column data from the data node again, and cache the data to the computing node.

[0041] For example, the compute node receives a query "select col0, col1, col2 from table", which involves the data of columns col0, col1, and col2 in the table. The compute node checks the cached data in the PKEY proxy storage engine and finds that the data of columns col0 and col1 are already in the cache. Assume that their PKEY version IDs are both 15df91230s. The data of column col2 is not in the PKEY proxy storage engine and needs to be obtained from the data node. After the last time the data of columns col0 and col1 were obtained from the data node, a new LevelFile was generated by the data node, and the metadata information stored in PKVersion changed. The version ID generated according to the current PKVersion is 261e7198ac. The data node sends the version ID and the data of column col2 to the compute node. The PKEY proxy storage engine compares the version IDs of columns col0, col1, and col2 and finds that the version IDs are inconsistent. It marks the cached data of columns col0 and col1 as invalid, obtains the data of columns col0, col1, and col2 from the data node again, uses these column data for query, and finally caches these column data in the PKEY proxy storage engine of the computing node.

[0042] Through the above method, the present invention can enable DolphinDB to give full play to the query advantages of PKEY on the computing nodes under the storage and computing separation architecture. To prove this effect, when all computing node caches are hit, the performance of computing node data caches carrying and not carrying data node sparse indexes and performing primary key point query (the advantageous query type of the PKEY storage engine) locally on the data node is compared.

[0043] Experimental environment:

[0044] CPU: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz

[0045] Memory: 512GiB (16*32 DDR4 3200MT / S)

[0046] Data scale: 60,000,000 data items, with a total size of approximately 6 GiB.

[0047] Compute node cache data carries data node cache query time Compute node cache data does not carry data node cache time Data node local query time consumption Primary key check 2.2ms 62.9ms 8.9ms

[0048] According to the experimental results, we can see that the column cache data in the PKEY proxy storage engine has the sparse index provided by the PKEY storage engine, so that the cache data on the computing node can take advantage of the PKEY storage engine (such as primary key point query in the experimental case).

[0049] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A query method for a proxy storage engine in a storage-computation separation architecture, applied to a computing node, characterized in that: Step 1: Use the proxy storage engine to check whether the required data is cached locally. If all the data is in the cache and the storage engine unique identifier of the data is consistent, execute step 2; if only part of the data is in the cache, execute step 3; Step 2: Use cached data for query; Step 3: Obtain the required data and the new storage engine unique identifier calculated based on the required data from the data node, and compare the new storage engine unique identifier with the storage engine unique identifier of the local corresponding cache data. If they are consistent, execute step 4; if they are inconsistent, execute step 5; Step 4: Use the required data to query, and cache the missing data and the corresponding storage engine unique identifier locally; Step 5: Obtain the required data from the data node for query, and cache the obtained data and its corresponding new storage engine unique identifier locally.

2. According to the query method of the proxy storage engine in the storage and computing separation architecture of claim 1, it is characterized by: The required data obtained from the data node is the column data stored in the columnar form by the storage engine of the data node. The obtained column data includes a set of column data segments. The column data segments are equipped with sparse indexes. When querying, the proxy storage engine filters the column data segments through the sparse indexes.

3. According to the query method of the proxy storage engine in the storage and computing separation architecture of claim 2, it is characterized by: The obtained column data is stored in column form on the data node through the primary key engine. The sparse index of the column data segment is a Bloom filter. Each column data segment of the primary key column contains a Bloom filter. The proxy storage engine performs primary key point query based on the Bloom filter.

4. According to the query method of the proxy storage engine in the storage and computing separation architecture of claim 1, it is characterized by: The storage engine unique identifier is obtained from the data node, and a version ID is calculated based on metadata corresponding to the required data stored in the storage engine metadata version of the data node.

5. A query method for a proxy storage engine in a storage-computation separation architecture, applied to a data node, characterized in that: The storage engine unique identifier corresponding to the calculated stored data is used to enable the computing node to cache the data and its corresponding storage engine unique identifier. When the proxy storage engine performs a query, it is checked whether the required data is in the cache and the storage engine unique identifier of the data is consistent. If so, the cached data is used for query. If only part of the data is in the cache, a new storage engine unique identifier is calculated based on the data required by the computing node to compare it with the storage engine unique identifier cached locally by the computing node. If they are consistent, the computing node uses the required data for query and caches the missing data and the corresponding storage engine unique identifier locally in the computing node. If they are inconsistent, the required data is sent to the computing node for query, and the computing node caches the acquired data and its corresponding new storage engine unique identifier locally.

6. According to the query method of the proxy storage engine in the storage and computing separation architecture of claim 5, it is characterized by: The storage engine stores data in columns at the data node. The column cache of the storage engine includes a set of column data segments. Each column data segment represents a storage unit of the storage engine. Each column data segment is equipped with a sparse index so that when the computing node queries, the proxy storage engine filters the column data segment through the sparse index.

7. According to the query method of the proxy storage engine in the storage and computing separation architecture of claim 6, it is characterized by: The storage unit includes a data storage unit located in a disk and a data storage unit located in a memory; The column data segments corresponding to the data storage units in the disk are arranged in the order of new and old, and the sparse indexes that have been stored in the column data segments of the data storage units in the disk can be directly reused; The data storage unit in the memory, if there is a data storage unit in the memory, its corresponding column data segment is arranged at the end.

8. According to the query method of the proxy storage engine in the storage and computing separation architecture of claim 6, it is characterized by: The storage engine is a primary key engine, and the sparse index of its column data segment is a Bloom filter. Each column data segment of the primary key column contains a Bloom filter, so that the proxy storage engine performs primary key point query based on the Bloom filter.

9. According to the query method of the proxy storage engine in the storage and computing separation architecture of claim 6, it is characterized by: The data node calculates the storage engine version ID based on the metadata of the storage unit in its current storage engine metadata version, and sends it to the compute node together with the column data to compare it with the storage engine version ID in the compute node's local cache.

10. A query system for a proxy storage engine under a storage-computing separation architecture, comprising a computing node and a data node, characterized in that: The computing node checks whether the required data is cached locally through the proxy storage engine. If the data is in the cache and the storage engine unique identifier of the data is consistent, the cached data is used for query; If only part of the data is in the cache, the required data and the corresponding new storage engine unique identifier are obtained from the data node, and the new storage engine unique identifier is compared with the storage engine unique identifier of the local corresponding cache data. If they are consistent, the required data is used for query, and the missing data and the corresponding storage engine unique identifier are cached locally; If they are inconsistent, the required data is obtained from the data node for query, and the obtained data and its corresponding new storage engine unique identifier are cached locally; The data node calculates the new storage engine unique identifier corresponding to the stored data, sends it to the computing node for comparison, and when the comparison is consistent, sends the required data to the computing node.

Images