Data stock computing architecture implementation system, method and equipment based on method calling and storage medium

Abstract

The invention discloses a data stock computing architecture implementation system, method and equipment based on method calling and a storage medium. The system comprises a computing engine and a storage engine, the calculation engine and the storage engine are in the same process and belong to a storage and calculation integrated model, and Driver layer data interaction is called by a CGO local method; the calculation engine and the storage engine are not in the same process and belong to a storage and calculation separation model, and Driver layer data interaction is called by an RPC remote method. The database architecture is flexible, the database dynamically provides services to the outside through a storage and calculation separation or storage and calculation integrated architecture, interaction between a calculation engine and a storage engine is simple, and a Driver layer provides multiple method calling modes to serve the database.

Description

Technical Field

[0001] The present invention relates to the technical field of database systems, and in particular to a method-call-based database computing architecture implementation system, method, device and storage medium. Background Art

[0002] The performance and reliability of a database system largely depends on the effective collaboration of its two core components, the storage engine and the computing engine. The storage engine is responsible for persistent storage and management of data, while the computing engine focuses on data processing tasks. There are two main architectural modes for the traditional combination of these two components: storage-computing integrated architecture and storage-computing separated architecture.

[0003] The integrated storage and computing architecture is a classic design in database systems, in which storage and computing functions are integrated in the same process. This tightly coupled approach brings the advantages of low latency, high data throughput, and easy management. However, with the growth of data scale and computing requirements, the integrated storage and computing architecture faces the problem of difficult horizontal expansion, and single point failures may affect the availability and reliability of the entire system. In order to solve the above problems, the storage-computing separation architecture decouples storage and computing functions and deploys them independently, which allows computing resources and storage resources to be expanded independently, improving the scalability, reliability, and fault tolerance of the system. With the popularization and development of cloud computing technology, the storage-computing separation architecture has become one of the mainstream development trends in the database field due to its flexibility and efficiency.

[0004] When the storage engine and the computing engine are separated, a driver layer needs to be introduced to be responsible for the data interaction between the two. However, the transmission of data in the network will add additional latency, which is a challenge for application scenarios that require high real-time performance. For example, when performing an adjacency information query in a graph database, the computing engine must first obtain the information of the initial point for processing, and then query its adjacency information. In this process, data needs to be transferred back and forth between the computing engine and the storage engine many times. Especially when the query involves multiple layers of adjacency relationships, the amount of data interaction will increase exponentially, which will significantly increase the network overhead and cause a significant increase in the cost of statement execution. In view of this, it is crucial to choose the right database architecture for different usage scenarios and data characteristics. To this end, it is necessary to implement the dynamic transformation of the storage engine and computing engine architecture online to ensure that the system can dynamically switch between the storage-computing integrated architecture and the storage-computing separated architecture according to actual needs, so as to achieve the best performance.

[0005] C / C++ is widely used in the development of storage engines for database systems due to its excellent performance and close proximity to the hardware layer. In contrast, computing engines focus more on user-oriented application logic and interact less with the underlying operating system, so the computing engine development process is usually more focused on concurrency, security, and ease of use. Based on these considerations, computing engines can be developed using the Go language. The Go language not only provides good development efficiency and good code maintainability, but also has excellent concurrency processing capabilities, making it one of the ideal choices for designing computing engines. For example, projects such as the distributed key-value storage system etcd, the columnar database management system ClickHouse, and the distributed relational database TiDB are written in C / C++ for their underlying core modules to ensure high-performance data storage and retrieval; while for the upper-level computing engine or auxiliary modules, they are written in the Go language to achieve a more efficient development process and better maintainability.

[0006] However, when using C / C++ and Go to develop a storage-computation separation database system, excessive data interaction costs will lead to degraded query performance when performing complex operations such as adjacency queries in graph databases. On the other hand, although stand-alone database systems can avoid such communication overhead, they still face many disadvantages brought by the storage-computation integration architecture, such as scalability limitations and single point failure risks. Therefore, designing a technical solution that can maintain high performance and effectively cope with complex query scenarios is crucial to improving the overall performance and reliability of database systems. Summary of the invention

[0007] Purpose of the invention: The present invention provides a system, method, device and storage medium for implementing a database computing architecture based on method calls. The database architecture is flexible, and the database dynamically provides services to the outside world with a storage-computing separation or storage-computing integration architecture. The interaction between the computing engine and the storage engine is simple, and the Driver layer provides a variety of method calling modes to serve the database.

[0008] Technical solution: The method call-based database computing architecture implementation system described in the present invention includes: a computing engine and a storage engine; the computing engine and the storage engine are in the same process, belonging to a storage-computing integrated model, and the driver layer data interaction is a CGO local method call; the computing engine and the storage engine are not in the same process, belonging to a storage-computing separation model, and the driver layer data interaction is an RPC remote method call.

[0009] Furthermore, different computing engines generate different execution plans based on the specific characteristics of the data, and the Driver layer selects different method call forms based on the information in the execution plan.

[0010] Furthermore, specific characteristics include the size of the data, the correlation between tables, and the distribution of the data.

[0011] Accordingly, a method for implementing a database computing architecture based on method calls includes the following steps:

[0012] Step 1, initial state: Assume that table A is initially created on node 1, that is, the data of table A is stored in storage engine 1. The user connects to computing engine 1 to query the data of table A. Computing engine 1 and storage engine 1 are deployed on the same node. Therefore, computing engine 1 enables table A to provide external services in the form of storage and computing integration.

[0013] Step 2: Conversion after data growth: After a period of time, when the amount of data in Table A grows to a point where it is not suitable to be stored on only one storage engine, Table A is dynamically redistributed, and part of the data in Table A is migrated to Storage Engine 2. At this time, the data in Table A is distributed in Storage Engine 1 and Storage Engine 2. The user still connects to Compute Engine 1 to query the data in Table A. Compute Engine 1 uses the CGO call method through the Driver layer to obtain the data in Table A on Storage Engine 1, and uses the RPC call method through the Driver layer to obtain the data in Table A on Storage Engine 2. Since Compute Engine 1 and Storage Engine 2 are in storage-computing separation mode, Compute Engine 1 is used to provide external services in the form of storage-computing separation.

[0014] Step 3, reverse conversion after the amount of data is reduced: On the contrary, when the amount of data in Table A is reduced to a certain threshold, the data of Table A on Storage Engine 2 is moved back to Storage Engine 1, so that Table A can provide services to the outside world in the form of integrated storage and computing again; when the data of Table A is stored in Storage Engine 1, only when the data of Table A is obtained through Compute Engine 1, Table A is in the integrated storage and computing design mode; when the data of Table A is obtained through Compute Engine 3, Table A presents a storage and computing separation design mode to the outside world, and the proxy forwarding should selectively forward statements to the computing engine with more efficient execution according to the data characteristics.

[0015] Furthermore, when a user sends an SQL statement to a computing engine to query the data of table A, in the stage of generating the execution plan tree of the SQL statement, the computing engine needs to determine the data distribution of table A based on the metadata of table A, and combine the computing engine's own metadata to determine the calling mode of querying leaf nodes in the execution plan tree.

[0016] Furthermore, the calling modes are divided into the following three types: a. Storage and computing integrated mode: The Driver layer uses CGO calls to query the local storage engine. This mode is suitable for the storage and computing integrated situation, that is, the computing engine and the storage engine are deployed on the same node; b. Storage and computing separation mode: The Driver layer uses RPC calls to query the remote storage engine. This mode is suitable for the situation where the computing engine is completely separated from the storage engine where Table A is located; c. Hybrid mode: When the data of Table A is distributed on different storage engines, the Driver layer will generate both CGO and RPC calling methods. In this case, part of the data is accessed through local CGO calls, and the other part is accessed through remote RPC calls.

[0017] Accordingly, a connection device for implementing a system for a database computing architecture based on method calls is characterized by comprising: one or more processors;

[0018] A storage device for storing one or more programs and user data;

[0019] When the one or more programs are executed by one or more processors, the one or more processors implement the method for implementing a database computing architecture based on method calls as described in any one of claims 4 to 6.

[0020] Correspondingly, a connection storage medium of a method call-based database computing architecture implementation system is characterized in that a computer program is stored thereon, and when the program is executed by a processor, it implements the method call-based database computing architecture implementation method as described in any one of claims 4 to 6.

[0021] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: the database architecture is flexible, and the database dynamically provides services to the outside in a storage-computing separation or storage-computing integration architecture. When the amount of data is small, the database provides the ultimate single-machine performance to the outside in the form of storage-computing integration. When the amount of data increases, it can be dynamically converted to a distributed mode. For some special scenarios such as graph adjacency queries, the hybrid mode can significantly reduce the amount of interaction between the computing engine and the storage engine on the network; the interaction between the computing engine and the storage engine is simple, and the Driver layer provides a variety of method call modes to serve the database, decoupling the transformation of the database service mode from the computing engine and the storage engine, and a variety of method call forms are concentrated in the Driver layer, which is conducive to simplifying the database module design. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic diagram of a three-node distributed hybrid deployment architecture of the present invention. DETAILED DESCRIPTION

[0023] like Figure 1As shown, a database computing architecture implementation system based on method calls includes: a computing engine and a storage engine; the computing engine and the storage engine are in the same process, which belongs to a storage-computing integrated model, and the driver layer data interaction is a CGO local method call; the computing engine and the storage engine are not in the same process, which belongs to a storage-computing separation model, and the driver layer data interaction is an RPC remote method call.

[0024] Different computing engines generate different execution plans based on the specific characteristics of the data, and the Driver layer selects different method call forms based on the information in the execution plan.

[0025] Furthermore, specific characteristics include the size of the data, the correlation between tables, and the distribution of the data.

[0026] Accordingly, a method for implementing a database computing architecture based on method calls includes the following steps:

[0027] Step 1, initial state: Assume that table A is initially created on node 1, that is, the data of table A is stored in storage engine 1. The user connects to computing engine 1 to query the data of table A. Computing engine 1 and storage engine 1 are deployed on the same node. Therefore, computing engine 1 enables table A to provide external services in the form of storage and computing integration.

[0028] Step 2: Conversion after data growth: After a period of time, when the amount of data in Table A grows to a point where it is not suitable to be stored on only one storage engine, Table A is dynamically redistributed, and part of the data in Table A is migrated to Storage Engine 2. At this time, the data in Table A is distributed in Storage Engine 1 and Storage Engine 2. The user still connects to Compute Engine 1 to query the data in Table A. Compute Engine 1 uses the CGO call method through the Driver layer to obtain the data in Table A on Storage Engine 1, and uses the RPC call method through the Driver layer to obtain the data in Table A on Storage Engine 2. Since Compute Engine 1 and Storage Engine 2 are in storage-computing separation mode, Compute Engine 1 is used to provide external services in the form of storage-computing separation.

[0029] Step 3, reverse conversion after the amount of data is reduced: On the contrary, when the amount of data in Table A is reduced to a certain threshold, the data of Table A on Storage Engine 2 is moved back to Storage Engine 1, so that Table A can provide services to the outside world in the form of integrated storage and computing again; when the data of Table A is stored in Storage Engine 1, only when the data of Table A is obtained through Compute Engine 1, Table A is in the integrated storage and computing design mode; when the data of Table A is obtained through Compute Engine 3, Table A presents a storage and computing separation design mode to the outside world, and the proxy forwarding should selectively forward statements to the computing engine with more efficient execution according to the data characteristics.

[0030] When a user sends an SQL statement to a certain computing engine to query the data of Table A, during the stage of generating the execution plan tree of the SQL statement, the computing engine needs to determine the data distribution of Table A based on the metadata of Table A, and combine the metadata of the computing engine itself to decide the call mode of the query leaf node in the execution plan tree.

[0031] The call modes are divided into the following three types: a. The integrated storage and computing mode: The Driver layer uses CGO calls to query the local storage engine. This mode is applicable to the integrated storage and computing situation, that is, the computing engine and the storage engine are deployed on the same node; b. The separated storage and computing mode: The Driver layer uses RPC calls to query the remote storage engine. This mode is applicable to the situation where the computing engine is completely separated from the storage engine where Table A is located; c. The hybrid mode: When the data of Table A is distributed on different storage engines, the Driver layer will generate both CGO and RPC call methods at the same time. In this case, a part of the data is accessed through local CGO calls, while the other part is accessed through remote RPC calls.

[0032] The present invention will be further described in detail below in combination with the implementation processes of the insert and query statements. For the convenience of description, only some relevant data structures and simple program processes of the present invention are given in the embodiments.

[0033] The client sends a statement to create a student table:

[0034]

[0035] Assume that the student table is created on Storage Engine 1 and Storage Engine 2 as shown in Figure 1 shown.

[0036] Embodiment 1: Insert data into the student table:

[0037] S101. The client inserts data into the student table. After the proxy receives the SQL sent by the client, it randomly distributes the SQL statement to Computing Engine 1 or 2 according to the distribution of the student table, and sends the SQL statement as follows:

[0038] insert into student(id,name,age)values

[0039] (1,'Zhang Yi',18),

[0040] (2,'Zhang Er',19),

[0041] (3,'Zhang San',20);

[0042] S102. Assume that the computing engine 1 receives the inserted SQL statement. The computing engine 1 obtains the distribution and distribution rules of the table through the metadata of the student table. Assume the data distribution is as follows:

[0043] Storage engine 1: (1, 'Zhang Yi', 18), (2, 'Zhang Er', 19)

[0044] Storage engine 2: (3, 'Zhang San', 20)

[0045] The simple execution plan tree generated by the computing engine 1 for the SQL statement is as follows:

[0046]

[0047] In the execution plan tree, ProduceResults is the root node, responsible for generating the returned results for the client. InsertNode is the leaf node, responsible for calling the Driver layer method to insert data into the storage engine. The Driver layer calls the cgo or rpc method according to the data mode of the InsertNode node. The pseudo-code is as follows:

[0048]

[0049] It should be noted that the InsertKeyValue method in the storage engine does not distinguish between cgo and rpc methods. It's just that the same method in the storage engine has two forms of local method call and remote method call in the driver layer. Storage engines 1 and 2 respectively receive the data distributed to themselves, persist the data, and return a response to the client to complete the entire insertion process.

[0050] Example 2: Query data in the student table:

[0051] S201. The client queries the data in the student table. The same proxy initially determines the distribution of the student table, and then shunts the SQL statement to the computing engine 1 or 2. The SQL statement sent is as follows:

[0052] select * from student where name = 'Zhang Yi'

[0053] S202. Assume that the computing engine 1 receives the query SQL. Then, the computing engine 1 can generate the following simple execution plan tree in combination with the metadata of the student table:

[0054]

[0055] ProduceResults is the root node responsible for generating client results, Filter is the left node for filtering, and TableScanNode is the leaf node responsible for scanning table records (no index is created by default, and table data is queried through full table scanning). The entire execution process is only slightly different in calling the Driver layer method in the TableScanNode node. The simple pseudo code for the Driver layer to call and query table data is as follows:

[0056]

[0057] The Filter node uses the query results returned by the child nodes (Iterator is the result iterator) and further filters the query results in combination with the where condition, and then returns the matching results to the upper parent node ProduceResults. The root node generates query information and returns it to the client to complete the entire query process.

Claims

1. A database computing architecture implementation system based on method call, characterized in that: include: Computing engine and storage engine: The computing engine and storage engine are in the same process, which belongs to the storage-computing integrated model. The data interaction at the Driver layer is a CGO local method call. The computing engine and storage engine are not in the same process, which belongs to the storage and computing separation model. The data interaction at the Driver layer is RPC remote method call.

2. The method call-based database computing architecture implementation system according to claim 1, characterized in that: Different computing engines generate different execution plans based on the specific characteristics of the data, and the Driver layer selects different method call forms based on the information in the execution plan.

3. The method call-based database computing architecture implementation system according to claim 1, characterized in that: Specific characteristics include the size of the data, the relationships between tables, and how the data is distributed.

4. A method for implementing a database computing architecture based on method call as claimed in claim 1, characterized in that: The steps include: Step 1, initial state: Assume that table A is initially created on node 1, that is, the data of table A is stored in storage engine 1. The user connects to computing engine 1 to query the data of table A. Computing engine 1 and storage engine 1 are deployed on the same node. Therefore, computing engine 1 enables table A to provide external services in the form of storage and computing integration. Step 2: Conversion after data growth: After a period of time, when the amount of data in Table A grows to a point where it is not suitable to be stored on only one storage engine, dynamically redistribute the data in Table A and migrate part of the data in Table A to Storage Engine 2. At this time, the data in Table A is distributed in Storage Engine 1 and Storage Engine 2. The user still connects to computing engine 1 to query the data of table A. Computing engine 1 uses the CGO call method through the Driver layer to obtain the data of table A on storage engine 1, and uses the RPC call method through the Driver layer to obtain the data of table A on storage engine 2. Since computing engine 1 and storage engine 2 are in storage-computing separation mode, computing engine 1 is used to provide external services for table A in the form of storage-computing separation. Step 3: Reverse conversion after the amount of data is reduced: On the contrary, when the amount of data in Table A is reduced to a certain threshold, the data of Table A on Storage Engine 2 is relocated to Storage Engine 1, so that Table A can provide services to the outside world in the form of storage and computing integration again; when the data of Table A is stored in Storage Engine 1, only when the data of Table A is obtained through Compute Engine 1, Table A is in the storage and computing integration design mode; When the data of Table A is obtained through computing engine 3, Table A presents a storage-computation separation design mode to the outside world. The proxy forwarding should selectively forward the statement to a computing engine with more efficient execution according to the data characteristics.

5. The method for implementing a database computing architecture based on method calls according to claim 4, characterized in that: When a user sends an SQL statement to a computing engine to query the data of table A, in the stage of generating the execution plan tree of the SQL statement, the computing engine needs to determine the data distribution of table A based on the metadata of table A, and combine the computing engine's own metadata to determine the call mode of querying leaf nodes in the execution plan tree.

6. The method for implementing a database computing architecture based on method calls according to claim 4, characterized in that: There are three calling modes: a. Storage-computing integrated mode: The driver layer uses CGO calls to query the local storage engine. This mode is suitable for storage-computing integrated situations, that is, the computing engine and storage engine are deployed on the same node; b. Storage and computing separation mode: The Driver layer uses RPC calls to query the remote storage engine. This mode is suitable for the situation where the computing engine is completely separated from the storage engine where Table A is located; c. Hybrid mode: When the data of Table A is distributed on different storage engines, the Driver layer will generate both CGO and RPC calls. In this case, part of the data is accessed through local CGO calls, and the other part is accessed through remote RPC calls.

7. A connection device for implementing a database computing architecture based on method calls, characterized in that: include: one or more processors; A storage device for storing one or more programs and user data; When the one or more programs are executed by one or more processors, the one or more processors implement the method for implementing a database computing architecture based on method calls as described in any one of claims 4 to 6.

8. A method-based database computing architecture implementation system connected to a storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the method for implementing a database computing architecture based on method call as described in any one of claims 4 to 6 is implemented.

Images