Proxy storage engine construction method of storage and calculation separation database system

Abstract

The invention discloses a proxy storage engine construction method of a storage and calculation separation database system, and belongs to the technical field of database systems, the database system comprises calculation nodes and data nodes, calculation tasks are obtained through the calculation nodes, and data source storage engine types required by the tasks are identified; constructing a group of proxy storage engines based on the corresponding data modals stored by the data nodes, acquiring required data from the proxy storage engines to complete task calculation according to the types of the storage engines, and if the proxy storage engines do not store the required data, sending data reading requests to the data nodes to obtain the required data to complete task calculation. Storing the required data in the proxy storage engine; and acquiring a data reading request through the data node, acquiring required data from the storage engine corresponding to the proxy storage engine, and returning the required data to the computing node. According to the method, the characteristics of different storage engines are utilized to cope with different query loads, and the advantages of multi-modal storage are exerted.

Description

Technical Field

[0001] The present invention relates to the technical field of database systems, and in particular to a method for constructing a proxy storage engine for a storage-computation separation database system. Background Art

[0002] In the storage-computing separation architecture of the database system DolphinDB, the cluster contains data nodes and computing nodes. Data nodes are responsible for writing and storing data; computing nodes are responsible for executing computing tasks submitted by users (i.e., SQL queries). Since computing nodes do not store data, data must be transferred from data nodes to computing nodes before computing is performed on them. In order to reduce the latency caused by data transmission, a layer of cache is usually added to the computing nodes to cache the data read from the data nodes locally on the computing nodes.

[0003] The database system DolphinDB supports multimodal storage engines, that is, DolphinDB implements multiple different storage engines for different usage scenarios, and the query load of each storage engine is different. For example, the TSDB (Time Series Database) engine is suitable for queries and aggregations for a single time series; the PKey (Primary Key) engine is suitable for single-point queries for primary keys. Under the storage-computing separation architecture, computing nodes also need to take advantage of multimodal storage engines. If the same memory data format is used in the computing engine to cache data in different modes and different storage engines, it will not only introduce significant overhead for data format conversion, but also occupy more memory space, reducing the efficiency of queries and calculations. Implementing an agent for each storage engine on the computing node can not only provide the capabilities of the storage engine, but also respond to each data mode in a differentiated manner, so that the performance of each mode can be maximized. Summary of the invention

[0004] In order to solve the shortcomings of the prior art and realize the purpose of giving full play to the advantages of multi-modal storage on computing nodes under the storage-computation separation architecture, the present invention adopts the following technical solutions: A method for constructing a proxy storage engine with storage and computing separation, applied to computing nodes; Build a computing engine and a cache layer. The cache layer includes a group of proxy storage engines. The proxy storage engines are built based on the corresponding data modes stored in the data nodes. The same interface is set for the proxy storage engines and the corresponding storage engines on the data nodes, so that the upper-layer computing engine can be seamlessly connected to the proxy storage engine without any modification (or with a small amount of modification). Since the proxy storage engine is built on demand based on the storage engine corresponding to the data node, it is only necessary to build the corresponding proxy storage engine in the computing node when the data node actually stores data of a certain mode. Otherwise, it is not necessary to build it. The computing engine obtains a computing task and identifies a data source storage engine type required for the task; The proxy storage engine obtains the required data from the proxy storage engine to complete the task calculation according to the data source storage engine type. If the proxy storage engine does not store the required data, a data read request is sent to the data node. The storage engine corresponding to the proxy storage engine is found from a group of storage engines of the data node through the data source storage engine type, and the required data is obtained from the proxy storage engine to complete the task calculation. The required data is then stored in the proxy storage engine. The data can be directly obtained from the proxy storage engine at the next cache hit.

[0005] Furthermore, the required data is time series data, including entities, attributes of entities that change over time, and timestamps. The computing task for time series data is to query the attributes of an entity within a certain time range, and the storage engine is a time series storage engine.

[0006] Furthermore, an index is created for the fields contained in the entity, and the read data request also includes the index to obtain the required data corresponding to the index from the data node. By utilizing the characteristics of the index, the cache granularity can be finer, thereby reducing the amount of data carried by the read data request and reducing the amount of data transmission.

[0007] Furthermore, the required data is obtained from the data node through remote procedure calls.

[0008] A method for constructing a proxy storage engine with storage and computing separation, applied to data nodes; A group of storage engines is constructed based on the corresponding data modes stored in the data nodes, and the same interface is set for the storage engines and the corresponding proxy storage engines in the computing node cache layer, so that the computing nodes can identify the data source storage engine type required for the computing task through their computing engines, and find the corresponding storage engine from a group of storage engines of the data nodes through the data source storage engine type, and send a data read request; According to the data read request, the required data is obtained from the storage engine and sent to the corresponding proxy storage engine, so that the computing node completes the computing task, and the proxy storage engine saves the required data, so that the computing node can directly obtain the required data from the proxy storage engine according to the data source storage engine type of the computing task.

[0009] Furthermore, the storage engine is a time series storage engine, and the required data stored is time series data, including entities, attributes of entities that change over time, and timestamps. The computing task for time series data is to query the attributes of an entity within a certain time range.

[0010] Furthermore, the obtained read data request includes an index created for the fields contained in the entity, and the required data is queried and returned based on the index. By utilizing the characteristics of the index, the cache granularity can be finer, thereby reducing the amount of data carried by the read data request and reducing the amount of data transmitted.

[0011] A database system with storage and computing separation, including computing nodes and data nodes; The computing node includes a computing engine and a cache layer, the cache layer includes a group of proxy storage engines, the proxy storage engine is constructed based on the corresponding data mode stored in the data node, the same interface is set for the proxy storage engine and the corresponding storage engine on the data node, the computing engine obtains the computing task and identifies the data source storage engine type required for the task; according to the storage engine type, the required data is obtained from the proxy storage engine to complete the task calculation, if the proxy storage engine does not store the required data, then the storage engine corresponding to the proxy storage engine is found from a group of storage engines of the data node according to the data source storage engine type, and a read data request is sent to it to obtain the required data to complete the task calculation, and the required data is stored in the proxy storage engine, and the data can be directly obtained from the proxy storage engine when the cache hits next time; The data node includes a group of storage engines, obtains a read data request, and finds required data from the storage engine corresponding to the proxy storage engine according to the data request, and returns it to the computing node.

[0012] Furthermore, the required data is time series data, including entities, attributes of entities that change over time, and timestamps. The computing task for time series data is to query the attributes of an entity within a certain time range. The storage engine is a time series storage engine. The read data request includes the data source number including the database name, table name, column name, and index; The index is created for each segment of the column contained in the entity. The corresponding segment is found from the storage engine through the index, making the query faster and more accurate, and making the granularity of the proxy storage engine cache finer, thereby reducing the amount of data carried by the read data request and reducing the amount of data transmission.

[0013] The advantages and beneficial effects of the present invention are: The present invention provides a proxy storage engine construction method for a storage-computation separation database system, which constructs a proxy storage engine corresponding to a data node storage engine on a computing node to cache data from different storage engines, and can utilize the characteristics of different storage engines to cope with different query loads, give play to the advantages of multimodal storage, and at the same time, fully utilize the characteristics of the storage engine to minimize the amount of data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the structure of the database system in the present invention. DETAILED DESCRIPTION

[0015] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0016] The cluster under the storage and computing separation architecture includes computing nodes and data nodes; the data nodes are responsible for writing and storing data, and the data nodes include a storage layer, which includes multiple storage engines; the computing nodes are responsible for executing the computing tasks submitted by users, and the computing nodes include a computing engine and a cache layer. The present invention implements a proxy storage engine for different storage engines on the cache layer of the computing engine. Figure 1 As shown, the present invention proposes a proxy storage engine construction method for a storage-computation separation database system, wherein storage engines A / B / C are implemented on data nodes, and corresponding proxy storage engines A / B / C are implemented on computing nodes. The proxy storage engine A on the computing node interacts with the storage engine A on the data node through RPC (Remote Procedure Call), which specifically includes the following steps: Step 1: Receive computing tasks.

[0017] The computing node receives the computing task submitted by the user, such as a SQL query statement. The computing node analyzes the task content and identifies the data source required for the task, including the storage engine type of the data source. Taking the time series storage engine as an example, it mainly stores time series data, which is a collection of data arranged in chronological order, such as sensor data, financial market data, network logs, etc. Time series data usually consists of three parts: tags: entities, such as machine IDs and stock codes; Metrics: properties of an entity that change over time, such as utilization, temperature, price; timestamp: timestamp.

[0018] The time series storage engine usually creates indexes on the fields (i.e., columns) contained in tags, so tags are also called SortKey.

[0019] SQL queries for time series data usually query the attributes of an entity within a certain time range, for example: SELECT * FROM t WHERE deviceId = 1 AND time BETWEEN 13:30 AND 15:00 Among them, deviceId is the SortKey.

[0020] Step 2: Read data from the proxy storage engine.

[0021] According to the identified storage engine type, the computing node reads the required data from the proxy storage engine on it. If the proxy storage engine has cached all the required data, jump to step 4, otherwise, execute step 3.

[0022] Step 3: Read data through remote procedure call RPC.

[0023] The proxy storage engine on the compute node sends a read data request to the storage engine on the data node through a remote procedure call (RPC), for example: fetchRemoteData(dbName, tableName, columns, others); dbName: database name of the data source; tableName: the table name of the data source; columns: the name of the column requested to be read; others: other information related to the storage engine. For example, for some storage engines, some filtering conditions can be pushed to the data nodes to reduce the amount of data transmission.

[0024] After receiving the remote procedure call RPC, the data node calls the storage engine interface, reads the required data, and returns the result to the computing node. The computing node caches the read data in the proxy storage engine, so that the next time the cache hits, it can directly read the data from the proxy storage engine without RPC to the data node to get the data.

[0025] Taking the time series storage engine as an example, when designing a proxy time series storage engine, it is usually necessary to retain the index capability. At the same time, using this feature, the cache granularity can be finer, such as a segment of a column, and each segment corresponds to a SortKey. Therefore, the others parameter of fetchRemoteData can carry the SortKey contained in the requested data, and the data node only needs to read the segments corresponding to the SortKey of the required column, thereby reducing the amount of data transmission. Taking the above SQL query as an example, the others parameter can contain the following information: SortKey{1}, indicating that only the segments with SortKey 1 need to be obtained.

[0026] Step 4: The proxy storage engine of the computing node returns the data in the cache (newly read or existing data or a mixture of both) to the upper-level computing engine.

[0027] The proxy storage engine on the compute node provides the same interface as the storage engine on the data node, so that the upper-layer compute engine can be seamlessly connected to the proxy storage engine without any modification (or with only a small amount of modification).

[0028] When the proxy storage engine reads data from the storage engine through remote procedure calls (RPCs), it can make full use of the characteristics of the storage engine and minimize the amount of data transmission.

[0029] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a proxy storage engine with storage and computing separation, applied to a computing node, characterized in that: Constructing a computing engine and a cache layer, wherein the cache layer includes a group of proxy storage engines, wherein the proxy storage engines are constructed based on corresponding data modalities stored in the data nodes, and the same interface is set for the proxy storage engines and the corresponding storage engines on the data nodes; The computing engine obtains a computing task and identifies a data source storage engine type required for the task; The proxy storage engine obtains the required data from the proxy storage engine to complete the task calculation according to the data source storage engine type. If the proxy storage engine does not store the required data, a data read request is sent to the data node, and a storage engine corresponding to the proxy storage engine is found from a group of storage engines of the data node through the data source storage engine type, and the required data is obtained from the proxy storage engine to complete the task calculation, and then the required data is stored in the proxy storage engine.

2. According to the method for constructing a proxy storage engine with storage and computation separation in claim 1, it is characterized by: The required data is time series data, including entities, attributes of entities that change over time, and timestamps. The computing task for time series data is to query the attributes of an entity within a certain time range, and the storage engine is a time series storage engine.

3. According to the method for constructing a proxy storage engine with storage and computation separation in claim 2, it is characterized by: An index is created for the fields included in the entity, and the data read request also includes the index, so as to obtain required data corresponding to the index from the data node.

4. The method for constructing a proxy storage engine with storage and computation separation according to claim 1, characterized in that: Obtain the required data from the data node through remote procedure calls.

5. A method for constructing a proxy storage engine with storage and computing separation, applied to a data node, characterized in that: A group of storage engines is constructed based on the corresponding data modes stored in the data nodes, and the same interface is set for the storage engines and the corresponding proxy storage engines in the computing node cache layer, so that the computing nodes can identify the data source storage engine type required for the computing task through their computing engines, and find the corresponding storage engine from a group of storage engines of the data nodes through the data source storage engine type, and send a data read request; According to the data read request, the required data is obtained from the storage engine and sent to the corresponding proxy storage engine, so that the computing node completes the computing task, and the proxy storage engine saves the required data, so that the computing node can directly obtain the required data from the proxy storage engine according to the data source storage engine type of the computing task.

6. The method for constructing a proxy storage engine with storage and computation separation according to claim 5, characterized in that: The storage engine is a time series storage engine, and the required data stored is time series data, including entities, attributes of entities that change over time, and timestamps. The computing task for time series data is to query the attributes of an entity within a certain time range.

7. The method for constructing a proxy storage engine with storage and computation separation according to claim 6, characterized in that: The obtained read data request contains an index created for the fields contained in the entity, and the required data is queried and returned according to the index.

8. A database system with storage and computing separation, comprising computing nodes and data nodes, characterized in that: The computing node includes a computing engine and a cache layer. The cache layer includes a group of proxy storage engines. The proxy storage engines are constructed based on the corresponding data modalities stored in the data nodes. The same interface is set for the proxy storage engines and the corresponding storage engines on the data nodes. The computing engine obtains the computing task and identifies the data source storage engine type required for the task. According to the storage engine type, the required data is obtained from the proxy storage engine to complete the task calculation. If the proxy storage engine does not store the required data, the storage engine corresponding to the proxy storage engine is found from a group of storage engines of the data node through the data source storage engine type, and a read data request is sent to the storage engine to obtain the required data to complete the task calculation, and the required data is stored in the proxy storage engine; The data node includes a group of storage engines, obtains a read data request, and finds required data from the storage engine corresponding to the proxy storage engine according to the data request, and returns it to the computing node.

9. A database system with storage and computation separation according to claim 8, characterized in that: The required data is time series data, including entities, attributes of entities that change over time, and timestamps. The computing task for time series data is to query the attributes of an entity within a certain time range. The storage engine is a time series storage engine. The read data request includes the data source number including the database name, table name, column name, and index; The index is created for each segment of the column included in the entity, and the corresponding segment is found from the storage engine through the index.

Images