A distributed database

Abstract

The present application provides a distributed database, which divides a single-machine database into three layers: the SQL node module, the management node module and the storage node module. In this way, data is dispersed on multiple nodes. After the SQL node module receives an SQL request, it obtains the data sharding location through the management node module. After one or more nodes in the storage node module write or obtain data, the data is summarized in the SQL node module, thereby realizing the function of distributed data processing of the openGauss database.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a distributed database. Background Art

[0002] As database technology develops rapidly, people have higher and higher requirements for database processing capabilities. In order to improve the processing performance of databases, distributed databases are increasingly widely used in people's daily lives. However, the existing OpenGauss database only has a centralized database, not a distributed database. Therefore, the existing OpenGauss database cannot meet the daily needs of users. Summary of the invention

[0003] The present application provides and implements a distributed database to realize the function of distributed data processing of the openGauss database.

[0004] The present application provides a distributed database, which includes an SQL node module, a management node module, and a storage node module;

[0005] The SQL node module is used to receive the SQL request sent by the client, perform SQL parsing and optimization processing on the SQL request, generate an execution request, and send the execution instruction to the management node module; generate an execution instruction according to the global transaction identifier and data shard position corresponding to the execution request returned by the management node module, and send the execution instruction to the storage node module; return the processing result of the execution instruction returned by the storage node module to the client;

[0006] The management node module is used to determine the global transaction identifier and the data sharding position corresponding to the execution request according to the execution request, and return the global transaction identifier and the data sharding position corresponding to the execution request to the SQL node module;

[0007] The storage node module is used to process the data stored in the storage node module according to the execution instruction to obtain the processing result of the execution instruction.

[0008] Optionally, the distributed database further includes: a Flink module;

[0009] The Flink module is used to respond to the analysis instruction and send the query request to the management node module; generate a query instruction according to the data shard position corresponding to the query request returned by the management node module, and send the query instruction to the storage node module; analyze the query result of the query instruction returned by the storage node module;

[0010] The management node module is further used to query the data shard position corresponding to the query request according to the query request, and return the data shard position corresponding to the query request to the Flink module;

[0011] The storage node module is used to query the data stored in the storage node module according to the query instruction to obtain the query result of the query instruction.

[0012] Optionally, the SQL node module is specifically used to: receive an SQL request sent by a client, perform grammatical and lexical analysis on the SQL request to obtain an SQL statement; determine whether the SQL statement is legal, if the SQL statement is illegal, directly return an error message to the client, if the SQL statement has no error, determine whether the SQL statement is a query statement; if the SQL statement is a query statement, determine whether the query data corresponding to the SQL statement is in the cache in the memory, if the query data corresponding to the SQL statement is in the cache in the memory, obtain the query data corresponding to the SQL statement from the cache, and return the query data corresponding to the SQL statement to the client; if the query data corresponding to the SQL statement is not in the cache in the memory, generate an execution request corresponding to the SQL statement, and send the execution instruction to the management node module.

[0013] Optionally, the SQL node module is further specifically used for: if the SQL statement is not a query statement, determining whether the SQL statement is a delete statement; if the SQL statement is a delete statement, accessing the management node module to obtain the transaction identifier corresponding to the SQL statement, and adding a delete mark to the transaction data; if the SQL statement is not a delete statement, determining that the SQL statement is an insert statement, accessing the management node module to obtain the transaction identifier corresponding to the SQL statement, and determining the position of the data slice to be inserted, determining whether the data in the data slice to be inserted is full, and if the data in the data slice to be inserted is full, accessing the management node module, and allocating a new data slice position in the management node module;

[0014] The storage node module is also used to insert data at the data slice position to be inserted using the Raft protocol, and return the insertion result to the SQL node module so that the SQL node module returns the insertion result to the client; if the inserted data slice position is row storage data, then after the data is inserted into the row storage data, the Raft Learner protocol is used to asynchronously copy the data into column storage data.

[0015] Optionally, the management node module is specifically used to allocate a global transaction identifier corresponding to the execution request according to the execution request, and to query the data shard position corresponding to the execution request based on the metadata dictionary information, and return the global transaction identifier and data shard position corresponding to the execution request to the SQL node module; the management node module is specifically used to query the data shard position corresponding to the query request based on the metadata dictionary information according to the query request, and return the data shard position corresponding to the query request to the Flink module.

[0016] Optionally, the SQL node module is specifically used to obtain the data shard position corresponding to the execution request returned by the management node module, determine whether the data shard position is stored in the same data node in the storage node module, and if the data shard position is stored in the same data node in the storage node module, generate an execution instruction, and send the execution instruction to the storage node module, so that the storage node module obtains the processing result of the execution instruction at the data node, and the SQL node module returns the processing result of the execution instruction to the client and stores it in the cache; if the data shard position is stored in multiple data nodes in the storage node module, generate an execution instruction, and send the execution instruction to the storage node module, so that the storage node module obtains the processing result of the execution instruction at the multiple data nodes in parallel, and the SQL node module returns the processing result of the execution instruction to the client and stores it in the cache;

[0017] The storage node module is also used to count the usage of each data slice and write the usage of each data slice into the management node module.

[0018] Optionally, the management node module is also used to periodically scan the usage of each data shard in the storage node module; and based on the usage and parameter threshold of each data shard, determine whether the data shard is overheated. If the data shard is not overheated, no processing is performed; if the data shard is overheated, one or more new shard locations are allocated to the data shard, and a split instruction and shard location are sent to the storage node module so as to split the data shard and migrate part of the data to the allocated one or more new shard locations.

[0019] Optionally, the Flink module is specifically used to: receive an SQL request sent by a client, perform grammatical and lexical analysis on the SQL request to obtain an SQL statement; determine whether the SQL statement is legal; if the SQL statement is illegal, directly return an error message to the client; if the SQL statement is legal, determine whether the query data corresponding to the SQL statement is in the cache in the memory; if the query data corresponding to the SQL statement is in the cache in the memory, obtain the query data corresponding to the SQL statement from the cache, and return the analysis result of the query data corresponding to the SQL statement to the client; if the query data corresponding to the SQL statement is not in the cache in the memory, generate the query request corresponding to the SQL statement, and send the query request to the management node module.

[0020] Optionally, the SQL node module includes several openGuass SQL nodes, and the storage node module includes several openGuass storage nodes.

[0021] Optionally, the types of the SQL request include: data query SQL request, data addition SQL request, data deletion SQL request and data modification SQL request.

[0022] It can be seen from the above technical scheme that the present application provides a distributed database, which includes an SQL node module, a management node module, and a storage node module; the SQL node module is used to receive an SQL request sent by a client, perform SQL parsing and optimization processing on the SQL request, generate an execution request, and send the execution instruction to the management node module; generate an execution instruction according to the global transaction identifier and data shard position corresponding to the execution request returned by the management node module, and send the execution instruction to the storage node module; return the processing result of the execution instruction returned by the storage node module to the client; the management node module is used to determine the global transaction identifier and data shard position corresponding to the execution request according to the execution request, and return the global transaction identifier and data shard position corresponding to the execution request to the SQL node module; the storage node module is used to process the data stored in the storage node module according to the execution instruction to obtain the processing result of the execution instruction. In this embodiment, the stand-alone database is divided into three layers: the SQL node module, the management node module and the storage node module. The data is scattered on multiple nodes. After the SQL node module receives the SQL request, it obtains the data shard location through the management node module. After one or more nodes in the storage node module write or obtain data, the data is summarized in the SQL node module, thereby realizing the distributed data processing function of the openGauss database.

[0023] The further effects of the above-mentioned non-conventional preferred manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 A schematic diagram of the structure of a distributed database provided in one embodiment of the present application;

[0026] Figure 2 A schematic diagram of the structure of a distributed database provided in one embodiment of the present application;

[0027] Figure 3 A schematic diagram of a distributed database process provided by an embodiment of the present application;

[0028] Figure 4 A schematic diagram of a distributed database process provided by an embodiment of the present application;

[0029] Figure 5 A schematic diagram of a distributed database process provided by an embodiment of the present application;

[0030] Figure 6 A schematic diagram of a distributed database process provided in one embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0032] Various non-limiting implementations of the present application are described in detail below in conjunction with the accompanying drawings.

[0033] See also Figure 1 , the distributed database may include a SQL node module (i.e., a SQL node cluster), a management node module (i.e., a management node cluster), and a storage node module (i.e., a storage node cluster). In one implementation, the distributed database may also include a Flink module (i.e., a Flink cluster).

[0034] like Figure 2 As shown, the SQL node module (i.e., SQL node cluster module) may include several SQL nodes. The SQL node module is a computing layer cluster composed of multiple computers, which may focus on configuring high-performance CPU and memory. The SQL node module may publish the interface of the SQL protocol to the outside, be responsible for accepting client connections, perform SQL parsing and optimization, and ultimately generate a distributed execution plan. The SQL node cluster module, this module is the computing layer, the SQL node publishes the interface of the SQL protocol to the outside, is responsible for accepting client connections, perform SQL parsing and optimization, and ultimately generate a distributed execution plan. After the SQL node receives the SQL command, it obtains the data location through the management node, writes or obtains data at multiple data nodes, and then summarizes it at the SQL node and returns it to the client.

[0035] like Figure 2 As shown, the management node module (i.e., the management node cluster module) may include several management nodes. The management node module is a management and coordination layer cluster composed of multiple computers, which can focus on configuring high-performance CPU and memory. The management node is responsible for management functions, storing metadata, and assigning transaction IDs to distributed transactions. In addition, the management node also automatically and intelligently decides whether to split hot data shards and balances workloads based on data shard usage statistics. Management node cluster module, this module is a scheduling and management layer. The management node is responsible for management functions, storing metadata, and assigning transaction IDs to distributed transactions. In addition, the management node also automatically and intelligently decides whether to split hot data shards and balances workloads based on data shard usage statistics. The management node returns the transaction ID and data shard meta information to the SQL node cluster module, and only transmits the data shard meta information to the Flink cluster module. The management node also allocates and returns data shard meta information to the data node to complete intelligent sharding and data writing.

[0036] like Figure 2 As shown, the storage node module (i.e., storage node cluster module) may include several row storage nodes and column storage nodes. The storage node module is a data node cluster composed of multiple computers, which may focus on configuring high-performance hard disks. The data node is responsible for providing both row and column storage data, providing intelligent data sharding and automatically maintaining multiple copies. The storage node cluster module is the data layer. The row storage node and the column storage node are responsible for providing both row storage data and column storage data, providing intelligent data sharding and automatically maintaining multiple copies. The data node cluster module returns data to the SQL cluster module. The data node cluster module will also send a request to the management node to obtain new data shard metadata such as location.

[0037] like Figure 2As shown in the figure, the Flink module (i.e., the Flink cluster module) can include a Flink management node and several execution nodes. The Flink module is a Flink cluster composed of multiple computers, which can focus on configuring high-performance CPU and memory. The Flink node completes the efficient and horizontally scalable distributed online analysis function by integrating the distributed stream computing technology of Flink SQL on the Apache Flink platform. In the Flink cluster module, the Flink node completes the efficient and horizontally scalable distributed online analysis function by integrating the distributed stream computing technology of FlinkSQL on the Apache Flink platform.

[0038] These four hardware clusters communicate with each other through high-speed network cards and networks to realize a highly available and highly scalable distributed online transaction processing database. The above four nodes can be horizontally expanded to form a cluster of multiple nodes, and the nodes communicate with each other to realize a highly available and highly scalable distributed online transaction processing database.

[0039] In one implementation, the SQL node module is used to receive an SQL request sent by a client, perform SQL parsing and optimization processing on the SQL request, generate an execution request, and send the execution instruction to the management node module; generate an execution instruction based on the global transaction identifier and data shard position corresponding to the execution request returned by the management node module, and send the execution instruction to the storage node module; return the processing result of the execution instruction returned by the storage node module to the client. The management node module is used to determine the global transaction identifier and data shard position corresponding to the execution request based on the execution request, and return the global transaction identifier and data shard position corresponding to the execution request to the SQL node module. The storage node module is used to process the data stored in the storage node module according to the execution instruction to obtain the processing result of the execution instruction.

[0040] The distributed database also includes: a Flink module; the Flink module is used to respond to the analysis instruction and send the query request to the management node module; generate a query instruction according to the data shard position corresponding to the query request returned by the management node module, and send the query instruction to the storage node module; analyze the query result of the query instruction returned by the storage node module; the management node module is also used to query the data shard position corresponding to the query request according to the query request, and return the data shard position corresponding to the query request to the Flink module; the storage node module is used to query the data stored in the storage node module according to the query instruction to obtain the query result of the query instruction.

[0041] In one implementation, the SQL node module may include several openGuass SQL nodes, and the storage node module may include several openGuass storage nodes. The types of SQL requests include: data query SQL request, data addition SQL request, data deletion SQL request, and data modification SQL request.

[0042] This embodiment is applicable to the case where a distributed database based on openGauss implements intelligent data sharding and has both online transaction processing and online analysis functions. Specifically, the method includes: dividing openGauss into four different nodes: management, SQL and data nodes, plus a Flink node that provides online analysis functions. The architecture can be implemented in software and hardware and integrated into the device.

[0043] For example, if Figure 3 As shown, in this embodiment, the following methods can be specifically implemented:

[0044] 1. The client sends data query, addition, deletion and modification requests to the openGauss SQL node cluster.

[0045] 2. After receiving the client request, the SQL node cluster performs grammatical lexical analysis. If there is an error, it directly returns the error and information to the client. If there is no error, it sends a request to the management node cluster to query the location of the data shard being operated.

[0046] 3. The management node cluster receives the SQL node cluster request. If a transaction is required, a global transaction ID is assigned. The metadata dictionary is queried and the data shard location that needs to be operated is returned to the SQL node cluster.

[0047] 4. After the SQL node cluster obtains the transaction ID and the required data shard location, it generates an execution plan. Based on the data shard location, it simultaneously sends requests to multiple data nodes to obtain data, or to add, delete, or modify data.

[0048] 5. According to the execution plan, the data node obtains data from the row store or column store and returns it to the SQL node cluster. If the data is added, deleted, or modified, after the openGauss row store data is operated, it will be asynchronously replicated to column store data using the Raft Learner protocol. Because it is asynchronous, it does not affect the performance of row store transactions.

[0049] 6. After receiving the data shards, the SQL node cluster calculates and summarizes the results according to the execution plan and returns them to the client.

[0050] 7. Flink cluster is a distributed online analysis function that can be efficiently and horizontally expanded by integrating the distributed stream computing technology of Flink SQL on the Apache Flink platform. Flink sends a request to the management node cluster to query the location of the data shard being operated.

[0051] 8. The management node cluster receives the Flink cluster request, queries the metadata dictionary, and returns the data shard location to be queried to the Flink cluster.

[0052] 9. After the Flink cluster obtains the data shard location, it generates an execution plan. According to the data shard location, it simultaneously sends requests to multiple data nodes to obtain data.

[0053] 10. According to the execution plan, the data node obtains data mainly from the column store, and can also obtain data from the row store when necessary, and returns it to the Flink cluster.

[0054] 11. After receiving the data shards, the Flink cluster calculates and summarizes the results according to the execution plan and returns them.

[0055] In one implementation, the SQL node module is specifically used to: receive an SQL request sent by a client, perform grammatical and lexical analysis on the SQL request to obtain an SQL statement; determine whether the SQL statement is legal, if the SQL statement is illegal, directly return an error message to the client, if the SQL statement has no error, determine whether the SQL statement is a query statement; if the SQL statement is a query statement, determine whether the query data corresponding to the SQL statement is in the cache in the memory, if the query data corresponding to the SQL statement is in the cache in the memory, obtain the query data corresponding to the SQL statement from the cache, and return the query data corresponding to the SQL statement to the client, if the query data corresponding to the SQL statement is not in the cache in the memory, generate an execution request corresponding to the SQL statement, and send the execution instruction to the management node module.

[0056] The SQL node module is also specifically used for: if the SQL statement is not a query statement, determining whether the SQL statement is a delete statement; if the SQL statement is a delete statement, accessing the management node module to obtain the transaction identifier corresponding to the SQL statement, and adding a delete mark to the transaction data; if the SQL statement is not a delete statement, determining that the SQL statement is an insert statement, accessing the management node module to obtain the transaction identifier corresponding to the SQL statement, and determining the position of the data slice to be inserted, determining whether the data in the data slice to be inserted is full, and if the data in the data slice to be inserted is full, accessing the management node module, and allocating a new data slice position in the management node module;

[0057] The storage node module is also used to insert data at the data slice position to be inserted using the Raft protocol, and return the insertion result to the SQL node module so that the SQL node module returns the insertion result to the client; if the inserted data slice position is row storage data (that is, the data is inserted into the row storage node), then after the data is inserted into the row storage data, the Raft Learner protocol is used to asynchronously copy the data into column storage data (that is, the data is inserted into the column storage node).

[0058] The management node module is specifically used to allocate a global transaction identifier corresponding to the execution request according to the execution request, and to query the data shard position corresponding to the execution request based on the metadata dictionary information, and return the global transaction identifier and data shard position corresponding to the execution request to the SQL node module; the management node module is specifically used to query the data shard position corresponding to the query request based on the metadata dictionary information according to the query request, and return the data shard position corresponding to the query request to the Flink module.

[0059] The SQL node module is specifically used to obtain the data sharding position corresponding to the execution request returned by the management node module, determine whether the data sharding position is stored in the same data node in the storage node module, and if the data sharding position is stored in the same data node in the storage node module, generate an execution instruction, and send the execution instruction to the storage node module, so that the storage node module obtains the processing result of the execution instruction at the data node, and the SQL node module returns the processing result of the execution instruction to the client and stores it in the cache; if the data sharding position is stored in multiple data nodes in the storage node module, generate an execution instruction, and send the execution instruction to the storage node module, so that the storage node module obtains the processing result of the execution instruction at the multiple data nodes in parallel, and the SQL node module returns the processing result of the execution instruction to the client and stores it in the cache;

[0060] The storage node module is also used to count the usage of each data slice and write the usage of each data slice into the management node module.

[0061] For example, if Figure 4 As shown, in this embodiment, the following methods can be specifically implemented:

[0062] 1. The client sends data query, addition, deletion and modification SQL requests to the openGauss SQL node cluster.

[0063] 2. After receiving the client request, the SQL node cluster performs grammatical analysis to determine whether the SQL statement is legal.

[0064] 3. If the SQL is illegal, directly return the error and information to the client.

[0065] 4. If there is no error in the SQL, determine whether this SQL statement is a query statement?

[0066] 5. If it is a query statement, determine whether the queried data is already in the cache in the memory.

[0067] 6. If the data is in the cache, get the data from the cache, summarize it through the calculation of the SQL node, and return the data to the client.

[0068] 7. If it is not in the cache, send a request to the management node cluster to query the location of the data shard being operated.

[0069] 8. The management node cluster receives the query request from the SQL node cluster, queries the metadata dictionary information, and returns the data shard location that needs to be operated to the SQL node cluster.

[0070] 9. After the SQL node cluster obtains the required data shard location, it determines whether the required data is stored in the same data node.

[0071] 10. If the data is stored in the same data node, according to the execution statistics, select row storage or column storage on one data node to obtain data and return it to the SQL node. The data node also counts the usage of each data slice and writes it to the management node. The calculation results are returned to the client and the data is placed in the cache.

[0072] 11. If there are multiple data nodes, based on the execution statistics, select row storage or column storage in parallel on multiple data nodes to obtain data and return it to the SQL node. The data node also counts the usage of each data slice and writes it to the management node.

[0073] 12. After receiving the data shards, the SQL cluster node calculates and summarizes the data to obtain the result, returns it to the client, and stores the data in the cache.

[0074] 13. In the judgment in 5, if it is not a query statement, determine whether the SQL is a delete statement.

[0075] 14. If it is a delete statement, access the management node cluster to obtain the transaction ID, and add a mark to the data to indicate deletion, waiting for the Vacuum process to automatically clean up later.

[0076] 15. If it is not a delete statement, it can only be an insert statement. The update statement is decomposed into a delete statement first and then an insert statement.

[0077] 16. Access the management node cluster to obtain the transaction ID and find the location where the data slice is to be inserted.

[0078] 17. Determine whether the data slice to be inserted is full?

[0079] 18. If the data is full, the data cluster node accesses the management cluster to allocate a new data slice location.

[0080] 19. The data cluster node inserts data and uses the Raft protocol to ensure that data insertion is successful on multiple nodes. The result is returned to the SQL cluster, which then returns it to the client.

[0081] 20. After inserting the openGauss row storage data, it will be asynchronously replicated to column storage data using the Raft Learner protocol. Because it is asynchronous, it does not affect the performance of row storage transactions.

[0082] In one implementation, the management node module is also used to periodically scan the usage of each data shard in the storage node module; and based on the usage and parameter threshold of each data shard, determine whether the data shard is overheated. If the data shard is not overheated, no processing is performed; if the data shard is overheated, one or more new shard locations are allocated to the data shard, and a split instruction and shard location are sent to the storage node module so as to split the data shard and migrate part of the data to the allocated one or more new shard locations.

[0083] For example, if Figure 5 As shown, in this embodiment, the following methods can be specifically implemented:

[0084] 1. In the previous example, the usage statistics of the data shards have been saved to the management node cluster.

[0085] 2. The management node cluster will scan statistical data regularly.

[0086] 3. Based on statistical data and modifiable parameter thresholds, determine whether the shard data is overheated, that is, does the read and write load mainly occur on a few data shards?

[0087] 4. If it is not overheating, do nothing.

[0088] 5. If overheating occurs, the management cluster assigns one or more new shard locations.

[0089] 6. Send the split instruction and shard location to the data node cluster.

[0090] 7. The data node splits and migrates part of the data to new shards of other data nodes, so that the read and write hot spots are distributed to multiple data nodes for parallel processing.

[0091] 8. The data node returns the new shard information to the management cluster for storage and later use.

[0092] In one implementation, the Flink module is specifically used to: receive an SQL request sent by a client, perform grammatical and lexical analysis on the SQL request to obtain an SQL statement; determine whether the SQL statement is legal, if the SQL statement is illegal, directly return an error message to the client, if the SQL statement is legal, determine whether the query data corresponding to the SQL statement is in the cache in the memory, if the query data corresponding to the SQL statement is in the cache in the memory, obtain the query data corresponding to the SQL statement from the cache, and return the analysis result of the query data corresponding to the SQL statement to the client, if the query data corresponding to the SQL statement is not in the cache in the memory, generate the query request corresponding to the SQL statement, and send the query request to the management node module.

[0093] For example, if Figure 6 As shown, in this embodiment, the following methods can be specifically implemented:

[0094] 1. The Flink cluster integrates the distributed stream computing technology of Flink SQL on the Apache Flink platform to complete efficient and horizontally scalable distributed online analysis functions.

[0095] 2. The Flink cluster management node receives and parses the SQL request and determines whether it is an error.

[0096] 3. If there is an error, directly return the error and information to the client.

[0097] 4. If there are no errors, determine whether the queried data is already in the cache in the memory.

[0098] 5. If the data is in the cache, get the data from the cache, aggregate it through the calculation of the Flink node, and return the data.

[0099] 6. If it is not in the cache, send a request to the management node cluster to query the location of the data shard being operated.

[0100] 7. The management node cluster receives the Flink cluster request, queries the metadata dictionary, and returns the data shard location to be queried to the Flink cluster.

[0101] 8. After the Flink cluster obtains the data shard location, it uses the Flink streaming distributed computing framework and generates an execution plan based on execution statistics. According to the data shard location, it simultaneously sends requests to multiple data nodes to obtain data.

[0102] 9. According to the execution plan, the data node obtains data mainly from the column store, and can also obtain data from the row store when necessary, and returns it to the Flink cluster.

[0103] 10. After receiving the data shards, the Flink cluster calculates and summarizes the results according to the execution plan and returns them. At the same time, the shard reading information is passed to the data node cluster for storage.

[0104] It can be seen from the above technical scheme that the present application provides a distributed database, which includes an SQL node module, a management node module, and a storage node module; the SQL node module is used to receive an SQL request sent by a client, perform SQL parsing and optimization processing on the SQL request, generate an execution request, and send the execution instruction to the management node module; generate an execution instruction according to the global transaction identifier and data shard position corresponding to the execution request returned by the management node module, and send the execution instruction to the storage node module; return the processing result of the execution instruction returned by the storage node module to the client; the management node module is used to determine the global transaction identifier and data shard position corresponding to the execution request according to the execution request, and return the global transaction identifier and data shard position corresponding to the execution request to the SQL node module; the storage node module is used to process the data stored in the storage node module according to the execution instruction to obtain the processing result of the execution instruction. In this embodiment, the stand-alone database is divided into three layers: the SQL node module, the management node module and the storage node module. The data is scattered on multiple nodes. After the SQL node module receives the SQL request, it obtains the data shard location through the management node module. After one or more nodes in the storage node module write or obtain data, the data is summarized in the SQL node module, thereby realizing the distributed data processing function of the openGauss database.

[0105] That is to say, the present invention can divide openGauss into four different nodes: management, SQL and data nodes, plus Flink nodes that provide online analysis functions; management nodes are responsible for management functions, storing metadata, and assigning transaction IDs for distributed transactions; SQL nodes publish the interface of the SQL protocol to the outside, are responsible for accepting client connections, executing SQL parsing and optimization, and finally generating distributed execution plans; data nodes are responsible for providing row and column storage data at the same time, providing intelligent data sharding and automatic maintenance of multiple copies; these four nodes can be horizontally expanded to form a cluster of multiple nodes, and each node communicates with each other to achieve a highly available and highly scalable distributed online transaction processing database. The above technical solution divides the stand-alone database into three layers, management, calculation and data are separated, and data is scattered on multiple nodes. After the SQL node obtains the SQL command, it obtains the data location through the management node, and after writing or obtaining data on multiple data nodes, it is aggregated on the SQL node, realizing the distributed online transaction processing database function that the openGauss database does not currently have; the Flink node completes the efficient and horizontally scalable distributed online analysis function by integrating the distributed stream computing technology of Flink SQL on the ApacheFlink platform.

[0106] It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. The device and system embodiments described above are merely schematic, and the units described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art can understand and implement it without paying creative work.

[0107] The above is only a preferred specific implementation method, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in this application should be included in the protection scope of this application. Therefore, the protection scope of this application should be based on the protection scope of the claims.

Claims

1. A distributed database, characterized in that: The distributed database is an openGauss database, and the distributed database includes an SQL node module, a management node module, and a storage node module; The SQL node module is used to receive the SQL request sent by the client, perform SQL parsing and optimization processing on the SQL request, generate an execution request, and send the execution request to the management node module; generate an execution instruction according to the global transaction identifier and data shard position corresponding to the execution request returned by the management node module, and send the execution instruction to the storage node module; Returning the processing result of the execution instruction returned by the storage node module to the client; The management node module is used to determine the global transaction identifier and the data sharding position corresponding to the execution request according to the execution request, and return the global transaction identifier and the data sharding position corresponding to the execution request to the SQL node module; The storage node module is used to process the data stored in the storage node module according to the execution instruction to obtain the processing result of the execution instruction; The distributed database also includes: a Flink module; The Flink module is used to respond to the analysis instruction and send a query request to the management node module; generate a query instruction according to the data shard position corresponding to the query request returned by the management node module, and send the query instruction to the storage node module; analyze the query result of the query instruction returned by the storage node module; The management node module is further used to query the data shard position corresponding to the query request according to the query request, and return the data shard position corresponding to the query request to the Flink module; The storage node module is used to query the data stored in the storage node module according to the query instruction to obtain the query result of the query instruction; The management node module is specifically used to allocate a global transaction identifier corresponding to the execution request according to the execution request, and to query the data shard position corresponding to the execution request based on the metadata dictionary information, and return the global transaction identifier and data shard position corresponding to the execution request to the SQL node module; the management node module is specifically used to query the data shard position corresponding to the query request based on the metadata dictionary information according to the query request, and return the data shard position corresponding to the query request to the Flink module.

2. The distributed database according to claim 1, characterized in that: The SQL node module is specifically used to: receive an SQL request sent by a client, perform grammatical and lexical analysis on the SQL request to obtain an SQL statement; determine whether the SQL statement is legal, if the SQL statement is illegal, directly return an error message to the client, if the SQL statement has no error, determine whether the SQL statement is a query statement; If the SQL statement is a query statement, determine whether the query data corresponding to the SQL statement is in the cache in the memory. If the query data corresponding to the SQL statement is in the cache in the memory, obtain the query data corresponding to the SQL statement from the cache and return the query data corresponding to the SQL statement to the client. If the query data corresponding to the SQL statement is not in the cache in the memory, generate an execution request corresponding to the SQL statement and send the execution instruction to the management node module.

3. The distributed database according to claim 2, characterized in that: The SQL node module is also specifically used to: if the SQL statement is not a query statement, determine whether the SQL statement is a delete statement; if the SQL statement is a delete statement, access the management node module to obtain the transaction identifier corresponding to the SQL statement, and add a delete flag to the transaction data; If the SQL statement is not a delete statement, determine that the SQL statement is an insert statement, access the management node module to obtain the transaction identifier corresponding to the SQL statement, determine the position of the data slice to be inserted, and judge whether the data in the data slice to be inserted is full. If the data in the data slice to be inserted is full, access the management node module and allocate a new data slice position in the management node module; The storage node module is also used to insert data at the data slice position to be inserted using the Raft protocol, and return the insertion result to the SQL node module so that the SQL node module returns the insertion result to the client; if the inserted data slice position is row storage data, then after the data is inserted into the row storage data, the Raft Learner protocol is used to asynchronously copy the data into column storage data.

4. The distributed database according to claim 1, characterized in that: The SQL node module is specifically used to obtain the data shard position corresponding to the execution request returned by the management node module, determine whether the data shard position is stored in the same data node in the storage node module, and if the data shard position is stored in the same data node in the storage node module, generate an execution instruction, and send the execution instruction to the storage node module, so that the storage node module obtains the processing result of the execution instruction at the data node, and the SQL node module returns the processing result of the execution instruction to the client and stores it in the cache; If the data sharding location is stored in multiple data nodes in the storage node module, an execution instruction is generated, and the execution instruction is sent to the storage node module, so that the storage node module obtains the processing result of the execution instruction in parallel in the multiple data nodes, and the SQL node module returns the processing result of the execution instruction to the client and stores it in the cache; The storage node module is also used to count the usage of each data slice and write the usage of each data slice into the management node module.

5. The distributed database according to claim 4, characterized in that: The management node module is also used to periodically scan the usage of each data shard in the storage node module; and based on the usage and parameter threshold of each data shard, determine whether the data shard is overheated. If the data shard is not overheated, no processing is performed; if the data shard is overheated, one or more new shard locations are allocated to the data shard, and a split instruction and shard location are sent to the storage node module so as to split the data shard and migrate part of the data to the allocated one or more new shard locations.

6. The distributed database according to claim 1, characterized in that: The Flink module is specifically used to: receive an SQL request sent by a client, perform grammatical and lexical analysis on the SQL request to obtain an SQL statement; determine whether the SQL statement is legal, if the SQL statement is illegal, directly return an error message to the client; if the SQL statement is legal, determine whether the query data corresponding to the SQL statement is in the cache in the memory; if the query data corresponding to the SQL statement is in the cache in the memory, obtain the query data corresponding to the SQL statement from the cache, and return the analysis result of the query data corresponding to the SQL statement to the client; if the query data corresponding to the SQL statement is not in the cache in the memory, generate the query request corresponding to the SQL statement, and send the query request to the management node module.

7. The distributed database according to any one of claims 1 to 6, characterized in that: The SQL node module includes several openGuass SQL nodes, and the storage node module includes several openGuass storage nodes.

8. The distributed database according to any one of claims 1 to 6, characterized in that: The types of SQL requests include: data query SQL request, data addition SQL request, data deletion SQL request and data modification SQL request.

Images