A distributed database implementation method and apparatus

Abstract

The application provides a distributed database implementation method and device. The method comprises the following steps: receiving a data query request sent by a client, performing protocol analysis on the data query request by a structured query layer contained in a data node, and obtaining corresponding target data from a storage engine layer contained in the data node; the storage engine layer is used for storing actual data and is located in the same program as the structured query layer; and the target data obtained from the data node is forwarded to the client. By designing the structured query layer and the storage engine layer in the same data node, each data node of the distributed database contains the structured query layer and the storage engine layer, can directly call an interface to access the distributed database, improves the data access efficiency in the distributed database, and improves the overall performance of the distributed database.

Description

Technical Field

[0001] This invention relates to the field of distributed database technology, specifically to a method and apparatus for implementing a distributed database. It also relates to an electronic device and a processor-readable storage medium. Background Technology

[0002] With the rapid development of computer technology, the application of relational databases and digital communication technologies is becoming more and more widespread. Faced with the ever-expanding business data, how to improve the efficiency of data query has become an urgent technical problem to be solved.

[0003] Currently, most distributed databases in existing technologies adopt distributed middleware solutions with database sharding and table partitioning, such as Mycat and MySQL Route. The drawbacks of distributed middleware are: data query request forwarding needs to be handled at the proxy layer; each database node needs to parse the Structured Query Language (SQL) protocol; the proxy layer needs to perform secondary merging of the result set; and regardless of which part of the data is accessed, the proxy layer and the database layer will interact over the network. Furthermore, later native distributed solutions, such as TiDB and MySQL Cluster, completely decouple the SQL layer and the data node layer, deploying them as separate processes. Requests are distributed at the structured query layer to retrieve data from different data nodes. While this design achieves relatively thorough decoupling, it has a flaw: existing distributed database solutions or patents, regardless of which part of the data is accessed, all data interactions (including data access, transaction commit / rollback) between the proxy node / SQL node and the data node will involve network access, resulting in low data access efficiency.

[0004] Therefore, how to design an efficient distributed database implementation scheme, minimize cross-network data access between internal nodes, and improve the overall efficiency of the distributed database has become an important research topic in this field. Summary of the Invention

[0005] Therefore, the present invention provides a distributed database implementation method and apparatus to solve the problem that the existing distributed database implementation schemes have high limitations, resulting in low data access efficiency.

[0006] In a first aspect, the present invention provides a method for implementing a distributed database, comprising:

[0007] Receive data query requests sent by the client;

[0008] The data query request is parsed using the structured query layer contained in the data node, and the corresponding target data is obtained from the storage engine layer contained in the data node.

[0009] The storage engine layer is used to store actual data and is located within the same program as the structured query layer.

[0010] The target data obtained from the data node will be forwarded to the client.

[0011] In one embodiment, obtaining the corresponding target data from the storage engine layer included in the data node specifically includes:

[0012] Based on the storage engine layer in the local data node, data sharding information is obtained from the management node. If, according to the data sharding information, it is determined that the data corresponding to the data query request is not entirely stored in the local data node, a remote data call request is sent from the storage engine layer contained in the local data node to the storage engine layer contained in the non-local data node to obtain the required remaining target data. The local target data in the storage engine layer of the local data node and the remaining target data are merged and processed to obtain the target data.

[0013] The non-local data node is a data node in a preset data node cluster that is used to store actual data, excluding the local data node.

[0014] In one embodiment, the distributed database implementation method further includes: sending the data query request to the data node based on a preset management node; wherein the management node is used to manage the data nodes in the distributed database, provide load balancing capabilities for the corresponding data node cluster, and store data sharding information corresponding to the data node cluster.

[0015] In one embodiment, each data node in the data node cluster is a separate process.

[0016] In one embodiment, each data node in the data node cluster comprises a structured query layer and a storage engine layer.

[0017] In one embodiment, the structured query layer contained in the data node performs protocol parsing on the data query request and obtains the corresponding target data from the storage engine layer contained in the data node, specifically including:

[0018] The structured query layer based on the data node performs protocol parsing on the data query request, distributes the data request using the storage engine layer of the data node, and obtains the corresponding target data from the storage engine layer contained in the data node; and / or,

[0019] The structured query layer based on data nodes performs protocol parsing and data request distribution for the data query requests, and obtains the corresponding target data from the storage engine layer of the data nodes.

[0020] Secondly, the present invention also provides a distributed database implementation apparatus, comprising:

[0021] The query request receiving unit is used to receive data query requests sent by the client.

[0022] The target data query unit is used to perform protocol parsing on the data query request based on the structured query layer contained in the data node, and obtain the corresponding target data from the storage engine layer contained in the data node.

[0023] The storage engine layer is used to store actual data and is located within the same program as the structured query layer.

[0024] The target data forwarding unit is used to forward the target data obtained from the data node to the client.

[0025] In one embodiment, the target data query unit is specifically configured to: obtain data sharding information from the management node based on the storage engine layer in the local data node; if it is determined from the data sharding information that the data corresponding to the data query request is not entirely stored in the local data node, then send a remote data call request to the storage engine layer in a non-local data node based on the storage engine layer contained in the local data node to obtain the required remaining target data; and merge the local target data in the storage engine layer of the local data node and the remaining target data to obtain the target data.

[0026] The non-local data node is a data node in a preset data node cluster that is used to store actual data, excluding the local data node.

[0027] In one embodiment, the distributed database implementation apparatus further includes: a query request sending unit, configured to send the data query request to the data node based on a preset management node; wherein the management node is configured to manage the data nodes in the distributed database, provide load balancing capabilities for the corresponding data node cluster, and store data sharding information corresponding to the data node cluster.

[0028] In one embodiment, each data node in the data node cluster is a separate process.

[0029] In one embodiment, each data node in the data node cluster comprises a structured query layer and a storage engine layer.

[0030] In one embodiment, the target data query unit is specifically used for:

[0031] The structured query layer based on the data node performs protocol parsing on the data query request, distributes the data request using the storage engine layer of the data node, and obtains the corresponding target data from the storage engine layer contained in the data node; and / or,

[0032] The structured query layer based on data nodes performs protocol parsing and data request distribution for the data query requests, and obtains the corresponding target data from the storage engine layer of the data nodes.

[0033] Thirdly, the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the distributed database implementation method as described in any of the preceding claims.

[0034] Fourthly, the present invention also provides a processor-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the distributed database implementation method as described in any of the preceding claims.

[0035] The distributed database implementation method described in this invention allows the structured query layer and storage engine layer to be designed on the same data node. This ensures that each data node of the distributed database contains both the structured query layer and the storage engine layer. If the data that the structured query layer needs to access happens to be in or partially in the storage engine layer of the data node, it can directly call the interface for access, without needing to go through network interaction as in other solutions. This improves the access efficiency of data within the distributed database, thereby effectively enhancing the overall performance of the distributed database. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A flowchart illustrating the distributed database implementation method provided in an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the structure of a distributed database provided in an embodiment of the present invention;

[0039] Figure 3This is a schematic diagram of the structure of a distributed database implementation device provided in an embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] The following is a detailed description of embodiments of the distributed database implementation method described in this invention. Figure 1 The diagram shown is a flowchart illustrating a distributed database implementation method provided in an embodiment of the present invention. The specific implementation process includes the following steps:

[0043] Step 101: Receive the data query request sent by the client.

[0044] In this embodiment of the invention, the structured query layer (i.e., the structured query language layer) and the storage engine layer (data layer) need to be set up on the same data node beforehand. This ensures that each data node in the distributed database contains both the SQL (Structured Query Language) layer and the data layer, with each data node operating as a separate process, rather than decoupling the SQL and data layers into different processes. The advantage of this setup is that if the target data accessed through the SQL layer happens to be located in the data layer of that data node, or if a portion of it is located in the data layer, the interface can be directly invoked for access, eliminating the need for network interaction as in other solutions. This improves the efficiency of data access within the distributed database.

[0045] Specifically, such as Figure 2As shown, a distributed database consists of a management node cluster and a data node cluster. The management nodes in the management node cluster record data sharding information, are used for load balancing, and manage the data nodes in the distributed database; they are stateless and highly available. Each data node in the data node cluster is a separate process, functionally consisting of an upper-layer structured query layer and a lower-layer storage engine layer. The SQL layer is stateless, used for SQL protocol parsing and optimization, and therefore does not require attention to the specific data storage location. When querying or modifying data in the local data node's Data layer, if the target data is not present or not entirely present in the local data node's Data layer, the Data layer forwards the request to the Data layers of other data nodes in the cluster for processing. For the SQL layer, the storage engine layer below the data node can be viewed as a single-machine database, used to store various actual data such as sharded tables and global tables.

[0046] In practical implementation, after receiving a data query request (i.e., an SQL request) from a client, the request can be sent to a data node in the data node cluster based on a preset management node or a third-party load balancing tool. The management node manages the data nodes in the distributed database, providing load balancing capabilities for the corresponding data node cluster; and stores the data sharding information (metadata information) corresponding to the data node cluster. Shard migration, splitting, and merging are all initiated from this node, and the data nodes only store the information in memory.

[0047] In addition, the management node is also used to store machine information of the data node cluster, while the data nodes only keep the information in memory; collect statistical information reported by the data nodes at regular intervals, determine the allocation scheme, and record it in the corresponding table; and provide monitoring functions, including the data volume of each machine, the overall sharding status, etc., which will not be elaborated here.

[0048] Step 102: Based on the structured query layer contained in the data node, perform protocol parsing on the data query request and obtain the corresponding target data from the storage engine layer contained in the data node. The storage engine layer stores the actual data and resides within the same program as the structured query layer.

[0049] In this step, the data query request is parsed via the structured query layer within the data node, and the data to be retrieved is directly sent to the internal Data layer. Data sharding information is then obtained from the management node based on the storage engine layer within the local data node. If the data sharding information indicates that all the data corresponding to the data query request is stored in the local data node, the API (Application Programming Interface) is directly invoked to access the data and retrieve the target data from the Data layer within that data node. This avoids network interaction as required by other distributed database solutions, thus improving the efficiency of data access within the distributed database. It should be noted that if the data sharding information indicates that not all the data corresponding to the data query request is stored in the local data node, a remote data retrieval request is further sent from the storage engine layer within the local data node to a storage engine layer outside the local data node to retrieve the remaining target data. The local target data in the storage engine layer of the local data node and the remaining target data are then merged to obtain the target data. Finally, the target data is returned to the structured query layer within the data node.

[0050] The non-local data nodes are data nodes in a pre-defined data node cluster that, excluding the local data nodes, are used to store actual data. Each data node in the data node cluster is a separate process. Each data node in the data node cluster comprises two parts: a structured query layer and a storage engine layer.

[0051] It should be noted that, in addition to being able to perform protocol parsing of the data query request based on the structured query layer of the data node, distribute the data request using the storage engine layer of the data node, and obtain the corresponding target data from the storage engine layer contained in the data node, this step can also perform protocol parsing and data request distribution of the data query request based on the structured query layer of the data node, and obtain the corresponding target data from the storage engine layer of the data node. No specific limitations are made here.

[0052] Step 103: Forward the target data obtained from the data node to the client.

[0053] In practice, the management node and each data node (data shard) form a high-availability cluster with one master and multiple backups, for example, by using the Raft algorithm protocol to ensure consistency and high availability.

[0054] For example, sharded table t1 has two columns, a and b. The table data is sharded according to the hash value of column a. Column a has an index, while column b does not. The cluster has two shards. Data node 1 stores data rows where a = 1, 3, 5, 7, 9, and data node 2 stores data rows where a = 2, 4, 6, 8, 10. When executing `select * from t1 where a = 2` on data node 1, the SQL layer of data node 1 parses the query and requests the data where a = 2 from its own Data layer. The Data layer of data node 1 queries the management node and learns that the data where a = 2 should be on data node 2. Therefore, the Data layer of data node 1 directly sends a request to the Data layer of data node 2, retrieves the relevant data, and returns it to the SQL layer of data node 1.

[0055] For example, taking table t1 as an example, when executing `select * from t1 where b>1` on data node 1, the SQL layer of data node 1 parses the query and finds that a full table scan is required. It requests data where b is greater than 1 from the data layer of its own node. After querying the management node, the data layer of its own node learns that data where b>1 may exist in data nodes 1 and 2. Therefore, the data layer of its own node sends a request to the data layer of data node 2 while simultaneously performing a query locally. It then merges the results of the two queries and returns them to the SQL layer of its own node.

[0056] The distributed database implementation method described in this invention allows the structured query layer and storage engine layer to be designed on the same data node. This ensures that each data node of the distributed database contains both the structured query layer and the storage engine layer. If the data that the structured query layer needs to access happens to be in or partially in the storage engine layer of the data node, it can directly call the interface for access, without needing to go through network interaction as in other solutions. This improves the access efficiency of data within the distributed database and thus effectively enhances the overall performance of the distributed database.

[0057] Corresponding to the distributed database implementation method provided above, this invention also provides a distributed database implementation apparatus. Since the embodiments of this apparatus are similar to the method embodiments described above, the description is relatively simple. For relevant details, please refer to the description in the method embodiment section above. The embodiments of the distributed database implementation apparatus described below are merely illustrative. Please refer to... Figure 3 The diagram shown is a structural schematic of a distributed database implementation device provided in an embodiment of the present invention. The distributed database implementation device of the present invention specifically includes the following parts:

[0058] The query request receiving unit 301 is used to receive data query requests sent by the client.

[0059] The target data query unit 302 is used to parse the data query request according to the protocol based on the structured query layer contained in the data node, and obtain the corresponding target data from the storage engine layer contained in the data node. The storage engine layer is used to store the actual data and is located within the same program as the structured query layer.

[0060] The target data forwarding unit 303 is used to forward the target data obtained from the data node to the client.

[0061] The distributed database implementation device described in this embodiment of the invention can design the structured query layer and the storage engine layer on the same data node, so that each data node of the distributed database contains both the structured query layer and the storage engine layer. If the data that the structured query layer needs to access happens to be in or partially in the storage engine layer of the data node, it can directly call the interface to access it, without having to go through network interaction as in other solutions. This improves the access efficiency of data within the distributed database, thereby effectively improving the overall performance of the distributed database.

[0062] Corresponding to the distributed database implementation method provided above, this invention also provides an electronic device. Since the embodiment of this electronic device is similar to the method embodiment described above, it is described simply. For relevant details, please refer to the description in the method embodiment section above. The electronic device described below is merely illustrative. Figure 4 The diagram shown is a schematic representation of the physical structure of an electronic device disclosed in an embodiment of the present invention. The electronic device may include a processor 401, a memory 402, and a communication bus 403, wherein the processor 401 and the memory 402 communicate with each other via the communication bus 403. The processor 401 can invoke logical instructions in the memory 402 to execute a distributed database implementation method. This method includes: receiving a data query request sent by a client; parsing the data query request based on a structured query layer contained in a data node, and obtaining corresponding target data from a storage engine layer contained in the data node; wherein the storage engine layer is used to store actual data and is located within the same program as the structured query layer; and forwarding the target data obtained from the data node to the client.

[0063] Furthermore, the logical instructions in the aforementioned memory 402 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] On the other hand, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a processor-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the distributed database implementation method provided in the above-described method embodiments, the method including: receiving a data query request sent by a client; performing protocol parsing on the data query request based on a structured query layer contained in a data node, and obtaining corresponding target data from a storage engine layer contained in the data node; wherein, the storage engine layer is used to store actual data and is located within the same program as the structured query layer; and forwarding the target data obtained from the data node to the client.

[0065] In another aspect, embodiments of the present invention also provide a processor-readable storage medium storing a computer program. When executed by a processor, the computer program implements the distributed database implementation method provided in the above embodiments. The method includes: receiving a data query request sent by a client; performing protocol parsing on the data query request based on a structured query layer contained in a data node; obtaining corresponding target data from a storage engine layer contained in the data node; wherein the storage engine layer is used to store actual data and is located within the same program as the structured query layer; and forwarding the target data obtained from the data node to the client.

[0066] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0067] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for implementing a distributed database, characterized in that, include: Receive data query requests sent by the client; The data query request is parsed by the structured query layer contained in the data node, and the structured query layer obtains the corresponding target data from the storage engine layer contained in the data node through an in-process call; wherein, each data node is a separate process, and the structured query layer and the storage engine layer are set on the same data node; The storage engine layer is used to store actual data and is located within the same program as the structured query layer. The target data obtained from the data node will be forwarded to the client; Specifically, obtaining the target data from the storage engine layer included in the data node includes: Based on the storage engine layer in the local data node, data sharding information is obtained from the management node. If, according to the data sharding information, it is determined that the data corresponding to the data query request is not entirely stored in the local data node, a remote data retrieval request is sent from the storage engine layer contained in the local data node to the storage engine layer contained in the non-local data node to obtain the required remaining target data. The local target data in the storage engine layer of the local data node and the remaining target data are merged and processed to obtain the target data. Herein, the non-local data node is a data node in the preset data node cluster other than the local data node used to store the actual data. If, based on the data sharding information, it is determined that all the data corresponding to the data query request is stored in the local data node, then the API interface is directly called to access it and obtain the corresponding target data from the storage engine layer inside the local data node.

2. The distributed database implementation method according to claim 1, characterized in that, Also includes: The data query request is sent to the data node based on the preset management node; wherein, the management node is used to manage the data nodes in the distributed database, provide load balancing capabilities for the corresponding data node cluster, and store the data sharding information corresponding to the data node cluster.

3. The distributed database implementation method according to claim 1, characterized in that, Each data node in the data node cluster is a separate process.

4. The distributed database implementation method according to claim 1, characterized in that, Each data node in the data node cluster consists of two parts: a structured query layer and a storage engine layer.

5. The distributed database implementation method according to claim 1, characterized in that, The structured query layer contained in the data node performs protocol parsing on the data query request and obtains the corresponding target data from the storage engine layer contained in the data node, specifically including: The structured query layer based on the data node performs protocol parsing on the data query request, distributes the data request using the storage engine layer of the data node, and obtains the corresponding target data from the storage engine layer contained in the data node; and / or, The structured query layer based on data nodes performs protocol parsing and data request distribution for the data query requests, and obtains the corresponding target data from the storage engine layer of the data nodes.

6. A distributed database implementation apparatus, characterized in that, include: The query request receiving unit is used to receive data query requests sent by the client. The target data query unit is used to parse the data query request according to the protocol based on the structured query layer contained in the data node, and the structured query layer obtains the corresponding target data from the storage engine layer contained in the data node through an in-process call; wherein, each data node is a separate process, and the structured query layer and the storage engine layer are set on the same data node; The storage engine layer is used to store actual data and is located within the same program as the structured query layer. A target data forwarding unit is used to forward the target data obtained from the data node to the client; Specifically, the target data query unit is used to: obtain data sharding information from the management node based on the storage engine layer in the local data node; if it is determined from the data sharding information that the data corresponding to the data query request is not entirely stored in the local data node, then send a remote data call request from the storage engine layer contained in the local data node to the storage engine layer contained in the non-local data node to obtain the required remaining target data; merge the local target data in the storage engine layer of the local data node and the remaining target data to obtain the target data; wherein, the non-local data node is a data node in a preset data node cluster other than the local data node used to store actual data; if it is determined from the data sharding information that the data corresponding to the data query request is entirely stored in the local data node, then directly call the API interface to access and obtain the corresponding target data from the storage engine layer inside the local data node.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the distributed database implementation method as described in any one of claims 1-5.

8. A processor-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the distributed database implementation method as described in any one of claims 1-5.

Images