Request processing method and device for distributed database, equipment, medium and program

Abstract

The embodiment of the invention discloses a request processing method and device for a distributed database, equipment, a medium and a program, and the method applied to a coordination node comprises the steps: receiving a database operation request submitted to the distributed database, and analyzing the database operation request to obtain a request analysis result; determining the processing data volume of the database operation request according to the request analysis result; determining a target execution node of the database operation request according to the processing data volume of the database operation request; wherein the target execution node comprises a client node, a coordination node or a working node; and generating an execution plan corresponding to the database operation request, so that the target execution node executes the execution plan corresponding to the database operation request. According to the technical scheme, the request processing efficiency of the distributed database is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of database, and particularly relate to a request processing method and device of distributed database, electronic equipment, storage medium and program. BACKGROUND

[0002] The distributed database is a database system which stores data in multiple nodes in a computer network and realizes data sharing and collaborative processing through a distributed architecture. It realizes high availability and horizontal expansion through data sharding and replication mechanism, while maintaining logical integrity, and users do not need to care about the specific physical location of data.

[0003] In the distributed database, the front and back ends are divided into a client driver, a coordinator node and a worker node, supporting UDF (User-Defined Function) or stored procedures in languages such as java or c++. Usually, the client driver transmits the requested SQL statement to the coordinator end, and the dynamic compilation and execution of the request are completed in the coordinator end or the worker end, and the business logic of the request will not be executed in the client driver. SUMMARY

[0004] Embodiments of the present application provide a request processing method and device of distributed database, electronic equipment, storage medium and program, which can reduce the network data transmission amount between nodes when the distributed database processes requests, and further reduce the load and latency of the request processing process of the distributed database, and improve the request processing efficiency of the distributed database.

[0005] According to an aspect of the present application, a request processing method of a distributed database is provided, applied to a coordinator node, comprising:

[0006] receiving a database operation request submitted to the distributed database, and parsing the database operation request to obtain a request parsing result;

[0007] determining a processing data amount of the database operation request according to the request parsing result;

[0008] determining a target execution node of the database operation request according to the processing data amount of the database operation request; wherein the target execution node comprises a client node, a coordinator node or a worker node;

[0009] generating an execution plan corresponding to the database operation request, so that the target execution node executes the execution plan corresponding to the database operation request.

[0010] According to another aspect of the present application, there is provided a request processing method of a distributed database, applied to a client node, comprising:

[0011] receiving an execution plan corresponding to a database operation request generated by a coordination node;

[0012] determining class identification information of the database operation request according to the execution plan corresponding to the database operation request;

[0013] in a case where it is determined that there is class information matching the class identification information in a local class loader, executing the execution plan corresponding to the database operation request.

[0014] According to another aspect of the present application, there is provided a request processing apparatus of a distributed database, configured in a coordination node, comprising:

[0015] a database operation request parsing module, configured to receive a database operation request submitted to a distributed database, and parse the database operation request to obtain a request parsing result;

[0016] a processing data amount determining module, configured to determine a processing data amount of the database operation request according to the request parsing result;

[0017] a target execution node determining module, configured to determine a target execution node of the database operation request according to the processing data amount of the database operation request; wherein the target execution node comprises a client node, a coordination node or a worker node;

[0018] an execution plan generating module, configured to generate an execution plan corresponding to the database operation request, so as to enable the target execution node to execute the execution plan corresponding to the database operation request.

[0019] According to another aspect of the present application, there is provided a request processing apparatus of a distributed database, configured in a client node, comprising:

[0020] an execution plan receiving module, configured to receive an execution plan corresponding to a database operation request generated by a coordination node;

[0021] a request class information determining module, configured to determine class information of the database operation request according to the execution plan corresponding to the database operation request;

[0022] an execution plan executing module, configured to, in a case where it is determined that there is class information of the database operation request in a local class loader, execute the execution plan corresponding to the database operation request.

[0023] According to another aspect of the present application, there is provided an electronic device, comprising:

[0024] at least one processor; and

[0025] a memory communicatively connected with the at least one processor; wherein

[0026] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the request processing method of the distributed database according to any one of the embodiments of the present application.

[0027] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the request processing method of the distributed database according to any one of the embodiments of the present application when executed by the processor.

[0028] According to another aspect of the present application, a computer program product is also provided, which comprises a computer program for implementing the request processing method of the distributed database according to any one of the embodiments of the present application when executed by a processor.

[0029] The embodiments of the present application can receive a database operation request submitted to a distributed database by a coordination node, analyze the database operation request to obtain a request analysis result, determine a processing data volume of the database operation request according to the request analysis result, and further determine a target execution node of the database operation request according to the processing data volume of the database operation request, which can be a client node, a coordination node or a worker node. Further, the coordination node generates an execution plan corresponding to the database operation request, so that the target execution node executes the execution plan corresponding to the database operation request. If the target execution node is a client node, the coordination node can send the execution plan corresponding to the database operation request to the client node. The client node can determine class identification information of the database operation request according to the execution plan corresponding to the database operation request, and execute the execution plan corresponding to the database operation request when it is determined that there is class information matching the class identification information in a local class loader. It can be seen that the above technical solution can implement the request processing process in the multiple nodes of the distributed database, reduce the network data transmission volume between the nodes when the distributed database processes the request, and further reduce the load and latency of the request processing process of the distributed database, thereby solving the problems of high network data transmission volume, load and latency when the existing distributed database processes the request, and improving the request processing efficiency of the distributed database.

[0030] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments description. Obviously, the drawings in the following description only show some embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative effort based on these drawings should also fall within the scope of the present application.

[0032] Figure 1 is a flow chart of a request processing method of a distributed database provided by the first embodiment of the present application;

[0033] Figure 2 is a structural schematic diagram of a distributed database provided by the first embodiment of the present application;

[0034] Figure 3 is a flow chart of dynamic compilation and execution of a request by a coordination node as a target execution node provided by the first embodiment of the present application;

[0035] Figure 4 is a flow chart of a request processing method of a distributed database provided by the second embodiment of the present application;

[0036] Figure 5 is a flow chart of dynamic compilation and execution of a request by a client node as a target execution node provided by the second embodiment of the present application;

[0037] Figure 6 is a schematic diagram of a request processing device of a distributed database provided by the third embodiment of the present application;

[0038] Figure 7 is a schematic diagram of a request processing device of a distributed database provided by the fourth embodiment of the present application;

[0039] Figure 8 is a structural schematic diagram of an electronic device provided by the fifth embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the person skilled in the art better understand the present application, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of the present application.

[0041] It should be noted that the terms "first", "second", "third", and "fourth" and the like in the description and claims of the application and above drawings of the application are used only to distinguish similar objects, and do not necessarily have to be used to describe a particular sequential or chronological order. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] Embodiment one

[0043] Figure 1 is a flowchart of a request processing method of a distributed database provided by the first embodiment of the application. The present embodiment can be applied to the case where the coordination node determines the target execution node of the database operation request according to the processing data volume of the database operation request. The method can be executed by a request processing device of the distributed database. The device can be realized by software and / or hardware, and can generally be integrated in an electronic device where the coordination node is located. The electronic device can be a terminal device or a server device. The present application does not limit the specific device type of the electronic device. Correspondingly, as shown in Figure 1 the method includes the following operations:

[0044] S110, receiving a database operation request submitted to the distributed database, and parsing the database operation request to obtain a request parsing result.

[0045] The database operation request can be a processing request submitted by a user to the distributed database, and can be a data storage request, or a data processing request initiated based on a UDF, etc. as long as it can process the data stored in the distributed database. The present application does not limit the request type and request content of the database operation request. It can be understood that the database operation request can be a SQL (Structured Query Language) form request.

[0046] In the embodiments of the present application, the distributed database can include client nodes, coordination nodes and worker nodes. The nodes of the distributed database are responsible for storing and processing part of data, and can be deployed on different physical devices such as servers, work together through a network to achieve high availability and load balancing; or in the case of resource constraints, the nodes of the distributed database can also be deployed on the same device; or the nodes of the distributed database can also be deployed in virtual machines or containers, etc. The embodiments of the present application do not limit the specific deployment mode of the nodes of the distributed database.

[0047] The distributed database can receive the SQL statement corresponding to the database operation request initiated by the user through the client node, and transmit it to the coordination node. After receiving the database operation request, the coordination node can first parse the database operation request, including lexical analysis, syntax analysis and semantic analysis, etc. parsing process, generating AST (Abstract Syntax Tree), so as to convert the database operation request from text form to a form easier to analyze and execute, and taking the generated AST as the request parsing result. After obtaining the request parsing result, it can also be checked to ensure the legality, rationality and compliance of the database operation request.

[0048] S120, determining the processing data amount of the database operation request according to the request parsing result.

[0049] After the coordination node parses the request parsing result of the database operation request, the dynamic compilation and execution process of the request is completed on the side of the coordination node or the side of the worker node, but further analysis is made on the request parsing result to determine the processing data amount involved in the database operation request. The processing data amount of the database operation request is different, and the type of the node processing the database operation request is also different.

[0050] In an optional embodiment of the present application, the determining the processing data amount of the database operation request according to the request parsing result can include: determining the input parameter data amount, the output parameter data amount and the intermediate pulling data amount of the database operation request according to the request parsing result; calculating the data amount sum of the input parameter data amount, the output parameter data amount and the intermediate pulling data amount.

[0051] The input parameter data amount can be the number of data input to the distributed database system. The output parameter data amount can be the number of data contained in the output parameter returned after the execution of the database operation request is completed. The intermediate pulling data amount can be the number of intermediate processing data pulled from the worker node. The intermediate processing data can be one or more rows of data of a database table, or the entire database table, and the embodiments of the present application do not limit the data type and data content of the intermediate processing data.

[0052] Specifically, the coordination node can respectively count the input parameter data volume, the output parameter data volume and the intermediate pulling data volume of the database operation request according to the request resolution result, so as to analyze the specific size of the data volume in each dimension. Meanwhile, the coordination node can also calculate the sum of the input parameter data volume, the output parameter data volume and the intermediate pulling data volume, so as to obtain the overall data volume involved in the database operation request.

[0053] S130, determining a target execution node of the database operation request according to the processing data volume of the database operation request; wherein the target execution node includes a client node, a coordination node or a worker node.

[0054] The target execution node can be a node that can currently process the database operation request.

[0055] Correspondingly, after determining the processing data volume information of the database operation request, the node that can currently process the database operation request can be selected as the target execution node from the client node, the coordination node or the worker node based on the processing data volume of the database operation request.

[0056] In an optional embodiment of the present application, the determination of the target execution node of the database operation request according to the processing data volume of the database operation request can include: determining the coordination node as the target execution node in the case that the sum of the data volume of the database operation request is less than or equal to a first set data volume threshold; determining the client node as the target execution node in the case that the sum of the data volume of the database operation request is greater than the first set data volume threshold, the input parameter data volume is greater than a second set data volume threshold, and the intermediate pulling data volume is less than a third set data volume threshold; determining the client node as the target execution node in the case that the sum of the data volume of the database operation request is greater than the first set data volume threshold, the output parameter data volume is greater than a fourth set data volume threshold, and the intermediate pulling data volume is less than the third set data volume threshold; and determining the worker node as the target execution node in the case that the sum of the data volume of the database operation request is greater than the first set data volume threshold, the input parameter data volume is less than or equal to the second set data volume threshold, the output parameter data volume is less than or equal to the fourth set data volume threshold, and the intermediate pulling data volume is greater than or equal to the third set data volume threshold.

[0057] The first set data volume threshold, the second set data volume threshold, the third set data volume threshold and the fourth set data volume threshold can be thresholds set according to actual needs, and the specific values of the above thresholds are not limited in the embodiments of the present application.

[0058] Optionally, the coordination node can employ corresponding rules and cost models through an internal optimizer to determine the target execution node of the database operation request according to the processing data amount of the database operation request. Specifically, the database operation request with a small total data amount can be directly executed at the coordination worker node. That is, if the total data amount of the database operation request is less than or equal to a first set data amount threshold, indicating that the overall data amount transmitted between different nodes through the network is small, the coordination node can be determined as the target execution node. If the input parameter data amount of the database operation request is large and the intermediate pulling data amount is small, that is, the total data amount of the database operation request is greater than the first set data amount threshold, the input parameter data amount is greater than a second set data amount threshold, and the intermediate pulling data amount is less than a third set data amount threshold, indicating that the data amount of the input parameter data to be processed at the client node side is large, the client node can be determined as the target execution node. If the output parameter data amount of the database operation request is large and the intermediate pulling data amount is small, that is, the total data amount of the database operation request is greater than the first set data amount threshold, the output parameter data amount is greater than a fourth set data amount threshold, and the intermediate pulling data amount is less than the third set data amount threshold, indicating that the data amount of the output parameter data to be processed at the client node side is large, the client node can be determined as the target execution node. If the output parameter data amount and the input parameter data amount of the database operation request are both small, but the intermediate pulling data amount is large, that is, the total data amount of the database operation request is greater than the first set data amount threshold, the input parameter data amount is less than or equal to the second set data amount threshold, the output parameter data amount is less than or equal to the fourth set data amount threshold, and the intermediate pulling data amount is greater than or equal to the third set data amount threshold, indicating that the data amount to be processed at the client node side is small and the intermediate data amount to be processed at the worker node is large, the worker node can be determined as the target execution node.

[0059] Optionally, when the worker node needs to be determined as the target execution node, the affinity of the intermediate pulling data can also be considered, that is, one or more worker nodes involving the largest intermediate pulling data amount are selected as the target execution node.

[0060] As can be seen, the above technical solution can analyze the processing data amount of the database operation request at different nodes in depth, and then select the node with a large processing data amount as the target execution node according to the size of the processing data amount distributed at different nodes, which can greatly reduce the data amount transmitted between the client node, the coordination node and the worker node through the network, and further reduce the load and request processing delay of the distributed database system.

[0061] S140, generating an execution plan corresponding to the database operation request, so that the target execution node executes the execution plan corresponding to the database operation request.

[0062] Specifically, the client node is an interface for a business system to connect the distributed database, which can be connected with the coordination node through an interface such as JDBC (Java Database Connectivity) or ODBC (Open Database Connectivity). There are generally multiple coordination nodes to maintain HA (High Availability) high availability. The coordination node can be conveniently scaled according to the system load of the distributed database. The coordination node is responsible for authentication, SQL parsing, and generation of an execution plan. Optionally, the execution plan can include a logical execution plan and a physical execution plan. The coordination node can also schedule multiple worker nodes to execute the physical execution plan, and then return data to the client node. The worker node is a computing and storage node of the distributed database, and all business data is stored on the worker node. At the same time, the worker node is responsible for completing the data calculation and interaction of the MPP (Massively Parallel Processing) operator.

[0063] Correspondingly, if the target execution node is the client node, the coordination node can send the execution plan corresponding to the database operation request to the client node to complete the dynamic compilation and execution of the database operation request on the client node side. If the target execution node is the coordination node, the coordination node can directly complete the dynamic compilation and execution of the database operation request according to the execution plan corresponding to the database operation request. If the target execution node is the worker node, the coordination node can split the execution plan to generate a sub-execution plan adapted to the worker node, and then distribute the split sub-execution plan to the corresponding worker node, while transmitting the metadata and parameter information required for execution to the worker node, and being responsible for coordinating the execution process of each worker node and collecting the execution results. Finally, the coordination node aggregates and merges the results returned by each worker node. In the process of executing the sub-execution plan, the coordination node monitors the running state of each worker node, processes the dependency relationship between the worker nodes, such as the data association order of different shards in the association query, to ensure that the sub-execution plan is executed in order according to the established logic.

[0064] Figure 2 is a structure diagram of a distributed database provided by an embodiment of the present application. In a specific example, as shown in FIG. 1, the distributed database includes a client node, a coordination node, and a worker node. Figure 2As shown, the embodiment of the present application optimizes and improves the client node in the distributed database, specifically by adding two components, UDF ClassLoader (a class loader specially used for loading user-defined function related classes and resources) and UDF Class Cache (a cache used for caching function classes or related resources) in the client node. Among them, the UDF Class Cache component on the client node side is used for executing database operation requests in the client node. When there is a database operation request directly scheduled in the client node, and there is no cache of the class of the request in the UDF Class Cache on the client node side, the component can obtain the class of the request from the coordination node end through the RPC (Remote Procedure Call Protocol, Remote Procedure Call Protocol) interface. The UDF ClassLoader on the client node side is a custom classLoader for the request for the JVM (Java Virtual Machine, Java Virtual Machine) in the client node, which is used to load the class of the request. Correspondingly, if the client node uses the JDBC driver, the JVM can directly use the UDF ClassLoader. If the client node uses the ODBC driver, it needs to call the JVM to define the execution of the UDF ClassLoader through the JNI (Java Native Interface, Java Native Interface).

[0065] As Figure 2As shown, the coordination node is provided with three components, UDF Metadata, UDF Class Cache and UDFClassLoader. Among them, the UDF Metadata is used to store the metadata of the request (including two request types of UDF and stored procedure), including the request definition and the binary class compiled from the request. The class of the request has a version number. In comparison, if the request is in the UDF format, the identification of a UDF request can be composed of "UDF package path + UDF name + version number". The UDF Class Cache is used to load the class of the request from the request metadata of the coordination node, and can be loaded into the UDF ClassLoader of the coordination node. The UDF Cache can be a Map data structure of kv key-value pairs, the key is the identification of the request, such as "UDF package path + UDF name + version number", and the value is the class binary data dynamically compiled from the request. The UDF ClassLoader is the custom ClassLoader of the coordination node JVM for the request, which is used to load the class of the request and execute the scheduling request.

[0066] As shown, Figure 2 As shown, the worker node is provided with two components, UDF Class Cache and UDF ClassLoader. Among them, the UDF Class Cache component can obtain the class of the request from the coordination node through the RPC interface, and load it into the UDF ClassLoader of the worker node. The UDF Class Cache is also a Map data structure of kv key-value pairs, the key is the identification of the request, such as "UDF package path + UDF name + version number", and the value is the class binary data dynamically compiled from the request. At the same time, the UDF Class Cache component of the worker node has a request metadata monitoring mechanism, which can update the class of the request in time when the request changes, and trigger the ClassLoader to reload. The UDF ClassLoader is the custom ClassLoader of the worker node JVM for the request, which is used to load the class of the request. If the worker node is implemented in C++, it needs to use JNI to call the request in the JVM ClassLoader.

[0067] It should be noted that the UDF ClassLoader component in all nodes can store class classes of all forms of database operation requests, such as class classes of UDF form requests or stored procedure form requests, and is not limited to class classes of UDF form requests. Similarly, the UDF Class Cache component in the client node and the worker node can obtain class classes of all forms of database operation requests from the coordination node end through an RPC interface, and load the class classes into the UDF ClassLoader of the corresponding node. The UDF Class Cache component of the coordination node can load class classes of all forms of database operation requests from the request metadata of the coordination node, and load the class classes into the UDF ClassLoader of the coordination node.

[0068] In summary, the request of the type of stored procedure or UDF can be compiled into a.class file of java, so that the database operation request can be executed on the client node, the coordination node and the worker node. The database operation request supports execution on the client node, and the client node stores a cache of the request class, which is loaded and isolated by using a custom ClassLoader.

[0069] Figure 3 FIG. 1 is a flowchart of a process of dynamic compilation and execution of a request by a coordination node as a target execution node according to an embodiment of the present application. In a specific example, as shown in FIG. 1, the process of dynamic compilation and execution of a request by a coordination node as a target execution node includes the following operations: Figure 3

[0070] Step 1: The client Driver sends a call stored procedure (an execution statement in a database) or a SQL execution request of a UDF to the Coordinator node, so as to forward the database operation request submitted to the distributed database to the Coordinator end.

[0071] Step 2: After receiving the database operation request, the Coordinator end parses the SQL syntax of the database operation request to generate an AST and performs verification.

[0072] Step 3: The Coordinator end generates an execution plan for processing of the database operation request, and determines the data volume of the database operation request. Specifically, the Coordinator end determines that the total data volume of the database operation request is less than or equal to a first set data volume threshold, and determines the node as a target execution node.

[0073] ​Step 4: The current database operation request is executed at the Coordinator side, and the Coordinator side checks the metadata and version of the database operation request. If the Coordinator side determines that the metadata and the corresponding version of the class class of the database operation request do not exist in the local UDF Class Cache, the class class of the database operation request is loaded from the request metadata of the Coordinator side.

[0074] Step 5: The Coordinator side further loads the class class of the database operation request into the UDFClassLoader, and starts the internal logic scheduling of the database operation request.

[0075] Step 6: The internal logic scheduling operation of the database operation request obtains intermediate processing data, and the Coordinator side calculates and pulls the intermediate processing data by a sub-execution plan operator to worker1.

[0076] Step 7: The operator task in the database operation request is executed on worker1.

[0077] Step 8: Worker1 returns the intermediate processing data required by the database operation request.

[0078] Step 9: The internal logic scheduling flow of the database operation request obtains intermediate processing data, and the Coordinator side calculates and pulls the intermediate processing data by a sub-execution plan operator to worker2, which can be executed in parallel with step 6.

[0079] Step 10: The operator task in the database operation request is executed on worker2.

[0080] Step 11: Worker2 returns the intermediate processing data required by the database operation request.

[0081] Step 12: The database operation request completes the entire internal logic call.

[0082] Step 13: The Coordinator side returns the execution result of the database operation request to the client Driver.

[0083] The embodiment of the application receives a database operation request submitted by a distributed database through a coordination node, analyzes the database operation request to obtain a request analysis result, determines a processing data volume of the database operation request according to the request analysis result, and further determines a target execution node of the database operation request according to the processing data volume of the database operation request. The target execution node can be a client node, a coordination node or a worker node. Further, the coordination node generates an execution plan corresponding to the database operation request, so that the target execution node executes the execution plan corresponding to the database operation request. If the target execution node is a client node, the coordination node can send the execution plan corresponding to the database operation request to the client node, so that the execution plan corresponding to the database operation request is executed at the client node. It can be seen that the above technical solution can realize the execution of the request processing process in the multi-end node of the distributed database, can reduce the network data transmission volume between the nodes when the distributed database processes the request, and further reduce the load and time delay of the request processing process of the distributed database, thereby solving the problems of high network data transmission volume, load and time delay when the existing distributed database processes the request, and improving the request processing efficiency of the distributed database.

[0084] Embodiment two

[0085] Figure 4 is a flowchart of a request processing method of a distributed database provided by the second embodiment of the application. The embodiment can be applicable to the case that the client node is used as the target execution node to execute the database operation request. The method can be executed by a request processing device of the distributed database. The device can be realized by software and / or hardware, and can be generally integrated in an electronic device where the client node is located. The electronic device can be a terminal device or a server device, and is used in cooperation with the client node and the worker node. The specific device type of the electronic device is not limited in the embodiment of the application. Correspondingly, as shown in Figure 4 , the method includes the following operations:

[0086] S410, receiving an execution plan corresponding to the database operation request generated by the coordination node.

[0087] When the optimizer in the coordination node determines that the target execution node of the database operation request is the client node according to the processing data volume of the database operation request, the execution plan corresponding to the database operation request can be sent to the client node, so that the dynamic compilation and execution process of the request is completed at the client node.

[0088] S420, determining the class identification information of the database operation request according to the execution plan corresponding to the database operation request.

[0089] The class identification information is used to identify the class corresponding to the version of the database operation request.

[0090] After the client node receives the execution plan corresponding to the database operation request, the class identification information of the database operation request can be determined, such as determining the class name and the corresponding version number of the database operation request and other related information.

[0091] S430, in the case that the class information matched with the class identification information exists in the local class loader, the execution plan corresponding to the database operation request is executed.

[0092] Correspondingly, after the client node determines the class identification information of the database operation request, it can query whether the class matched with the class identification information exists in the local class loader, i.e., UDF ClassLoader. Specifically, the client node can query whether the class with the corresponding version number exists in the local class loader according to the class name and the corresponding version number of the database operation request and other related information. If the client node determines that the class information matched with the class identification information of the database operation request exists in the local class loader, and the class matched with the version number of the database operation request exists in the local class loader, the execution plan corresponding to the database operation request can be directly executed.

[0093] In an optional embodiment of the present application, the above method can further include: in the case that the class information matched with the class identification information does not exist in the local class loader, and the class information matched with the class identification information exists in the local class information cache, loading the class information matched with the class identification information stored in the local class information cache into the local class loader, and executing the execution plan corresponding to the database operation request; in the case that the class information matched with the class identification information does not exist in the local class loader and the local class information cache, requesting the class information matched with the class identification information from the coordination node according to the class identification information, loading the class information matched with the class identification information into the local class information cache, further loading the class information matched with the class identification information stored in the local class information cache into the local class loader, and executing the execution plan corresponding to the database operation request.

[0094] The local class information cache is also called local UDF Class Cache.

[0095] Figure 5 is a flowchart of a process of a client node as a target execution node completing dynamic compilation and execution of a request provided by the second embodiment of the present application. In a specific example, as shown in Figure 5 the process of the client node as the target execution node completing dynamic compilation and execution of the request includes the following operations:

[0096] Step 1: The client Driver sends a SQL execution request of a call procedure or a UDF to the Coordinator node, so as to forward the database operation request submitted to the distributed database to the Coordinator end.

[0097] Step 2: After receiving the database operation request, the Coordinator end parses the SQL syntax of the database operation request to generate an AST and performs a check.

[0098] Step 3: The Coordinator end generates an execution plan for processing the database operation request and determines the data volume of the database operation request. Specifically, the Coordinator end determines that the total data volume of the database operation request is greater than a first set data volume threshold, the input data volume is greater than a second set data volume threshold, and the intermediate pulling data volume is less than a third set data volume threshold; or the Coordinator end determines that the total data volume of the database operation request is greater than the first set data volume threshold, the output data volume is greater than a fourth set data volume threshold, and the intermediate pulling data volume is less than the third set data volume threshold, and determines the client node as a target execution node.

[0099] Step 4: The Coordinator end returns the execution plan scheduling information of the database operation request to the client Driver.

[0100] Step 5: The client Driver obtains the version information of the database operation request and checks whether the class corresponding to the version of the database operation request exists in the UDF ClassLoader. If it exists, the step 11 is executed, otherwise, step 6 is executed.

[0101] Step 6: The client Driver checks whether the class corresponding to the version of the database operation request exists in the UDF Class Cache. If it exists, the step 10 is executed, otherwise, step 7 is executed.

[0102] Step 7: The client Driver pulls the class based on the identifier of the database operation request to the Coordinator node.

[0103] Step 8: The Coordinator node returns the class binary to the client Driver.

[0104] Step 9: The client Driver loads the pulled class into the local UDF Class Cache.

[0105] Step 10: Client Driver further loads the calss class in the local UDF Class Cache into the UDF ClassLoader of the JVM.

[0106] Step 11: Client Driver starts to schedule the business logic of the database operation request.

[0107] Step 12: The internal logic of the database operation request of the Client Driver schedules the operation to obtain intermediate processing data from the Coordinator side.

[0108] Step 13: The Coordinator side obtains intermediate processing data from worker1.

[0109] Step 14: The operator task in the internal database operation request is executed on worker1.

[0110] Step 15: The intermediate processing data of the database operation request executed on worker1 is returned to the Coordinator side.

[0111] Step 16: The Coordinator side obtains intermediate processing data from worker2, which can be parallel to step 13.

[0112] Step 17: The operator task in the internal database operation request is executed on worker2.

[0113] Step 18: The intermediate processing data of the database operation request executed on worker2 is returned to the Coordinator side.

[0114] Step 19: The Coordinator side summarizes the intermediate processing data and sends the internal execution logic of the database operation request of the Client Driver.

[0115] Step 20: The logical operation in the database operation request directly generates a result set on the client node and returns to the business side.

[0116] In actual business processing, packet capture can be performed on the client node. If a large number of result sets are generated by the database operation request during the execution of the database operation request, and the data traffic of the client node and the coordination node is small, it indicates that the database operation request is directly executed on the client node. Alternatively, the network traffic of the client node and the coordination node can be monitored. If a large number of result sets are generated by the database operation request during the execution of the database operation request, and the data traffic of the client node and the coordination node is small, it indicates that the database operation request is directly executed on the client node.

[0117] The embodiment of the application receives a database operation request submitted to a distributed database through a coordination node, analyzes the database operation request to obtain a request analysis result, determines a processing data volume of the database operation request according to the request analysis result, and further determines a target execution node of the database operation request according to the processing data volume of the database operation request. The target execution node can be a client node, a coordination node or a worker node. Further, the coordination node generates an execution plan corresponding to the database operation request, so that the target execution node executes the execution plan corresponding to the database operation request. If the target execution node is a client node, the coordination node can send the execution plan corresponding to the database operation request to the client node. The client node can determine class identification information of the database operation request according to the execution plan corresponding to the database operation request, and execute the execution plan corresponding to the database operation request when it is determined that there is class information matching the class identification information in the local class loader. It can be seen that the above technical solution can realize the execution of the request processing process in the multi-end node of the distributed database, reduce the network data transmission volume between the nodes when the distributed database processes the request, and further reduce the load and time delay of the request processing process of the distributed database, thereby solving the problems of high network data transmission volume, load and time delay when the existing distributed database processes the request, and improving the request processing efficiency of the distributed database.

[0118] It should be noted that the related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data comply with relevant laws, regulations and standards in relevant regions.

[0119] It should be noted that any arrangement and combination of the technical features among the above embodiments also belong to the protection scope of the present application.

[0120] Embodiment three

[0121] Figure 6 is a schematic diagram of a request processing device of a distributed database provided by the third embodiment of the application, which is configured in a coordination node, as Figure 6 shown, the device comprises a database operation request analysis module 610, a processing data volume determination module 620, a target execution node determination module 630 and an execution plan generation module 640, wherein:

[0122] The database operation request analysis module 610 is configured to receive a database operation request submitted to a distributed database, and analyze the database operation request to obtain a request analysis result.

[0123] The processing data amount determination module 620 is configured to determine a processing data amount of the database operation request according to the request resolution result.

[0124] The target execution node determination module 630 is configured to determine a target execution node of the database operation request according to the processing data amount of the database operation request, wherein the target execution node comprises a client node, a coordination node or a worker node.

[0125] The execution plan generation module 640 is configured to generate an execution plan corresponding to the database operation request, so that the target execution node executes the execution plan corresponding to the database operation request.

[0126] Optionally, the processing data amount determination module 620 is further configured to determine an input parameter data amount, an output parameter data amount and an intermediate pulling data amount of the database operation request according to the request resolution result, and calculate a data amount sum of the input parameter data amount, the output parameter data amount and the intermediate pulling data amount.

[0127] Optionally, the target execution node determination module 630 is further configured to: in a case where the data amount sum of the database operation request is less than or equal to a first set data amount threshold, determine the coordination node as the target execution node; in a case where the data amount sum of the database operation request is greater than the first set data amount threshold, the input parameter data amount is greater than a second set data amount threshold, and the intermediate pulling data amount is less than a third set data amount threshold, determine the client node as the target execution node; in a case where the data amount sum of the database operation request is greater than the first set data amount threshold, the output parameter data amount is greater than a fourth set data amount threshold, and the intermediate pulling data amount is less than the third set data amount threshold, determine the client node as the target execution node; and in a case where the data amount sum of the database operation request is greater than the first set data amount threshold, the input parameter data amount is less than or equal to the second set data amount threshold, the output parameter data amount is less than or equal to the fourth set data amount threshold, and the intermediate pulling data amount is greater than or equal to the third set data amount threshold, determine the worker node as the target execution node.

[0128] The embodiment of the present application receives a database operation request submitted by a distributed database through a coordination node, analyzes the database operation request to obtain a request analysis result, determines a processing data volume of the database operation request according to the request analysis result, and further determines a target execution node of the database operation request according to the processing data volume of the database operation request. The target execution node can be a client node, a coordination node or a worker node. Further, the coordination node generates an execution plan corresponding to the database operation request, so that the target execution node executes the execution plan corresponding to the database operation request. If the target execution node is a client node, the coordination node can send the execution plan corresponding to the database operation request to the client node, so that the client node executes the execution plan corresponding to the database operation request. It can be seen that the above technical solution can realize the execution of the request processing process in the multi-end node of the distributed database, can reduce the network data transmission volume between the nodes when the distributed database processes the request, and further reduce the load and time delay of the request processing process of the distributed database, thereby solving the problems of high network data transmission volume, load and time delay when the existing distributed database processes the request, and improving the request processing efficiency of the distributed database.

[0129] The request processing device of the distributed database described above can execute the request processing method of the distributed database performed by the coordination node provided by any embodiment of the present application, has the function modules and beneficial effects corresponding to the execution method. The technical details not described in detail in the present embodiment can be referred to the request processing method of the distributed database performed by the coordination node provided by any embodiment of the present application.

[0130] Embodiment four

[0131] Figure 7 is a schematic diagram of a request processing device of a distributed database provided by an embodiment of the present application. The device is configured in a client node, as shown in Figure 7 The device comprises an execution plan receiving module 710, a request class information determining module 720 and an execution plan executing module 730, wherein:

[0132] The execution plan receiving module 710 is configured to receive an execution plan corresponding to a database operation request generated by a coordination node;

[0133] The request class information determining module 720 is configured to determine class information of the database operation request according to the execution plan corresponding to the database operation request;

[0134] The execution plan executing module 730 is configured to execute the execution plan corresponding to the database operation request in the case that the class information of the database operation request exists in the local class loader.

[0135] Optionally, the apparatus further comprises a first information loading module configured to, in a case where it is determined that the class information matching the class identifier information does not exist in the local class loader and the class information matching the class identifier information exists in the local class information cache, load the class information matching the class identifier information stored in the local class information cache into the local class loader, and execute the execution plan corresponding to the database operation request; and a second information loading module configured to, in a case where it is determined that the class information matching the class identifier information does not exist in the local class loader and the local class information cache, request the class information matching the class identifier information from the coordination node according to the class identifier information, load the class information matching the class identifier information into the local class information cache, further load the class information matching the class identifier information stored in the local class information cache into the local class loader, and execute the execution plan corresponding to the database operation request.

[0136] The embodiment of the present application can receive the database operation request submitted by the distributed database through the coordination node, analyze the database operation request to obtain a request analysis result, determine the processing data volume of the database operation request according to the request analysis result, and further determine the target execution node of the database operation request according to the processing data volume of the database operation request. The target execution node can be a client node, a coordination node or a worker node. Further, the coordination node generates an execution plan corresponding to the database operation request, so that the target execution node executes the execution plan corresponding to the database operation request. If the target execution node is a client node, the coordination node can send the execution plan corresponding to the database operation request to the client node. The client node can determine the class identifier information of the database operation request according to the execution plan corresponding to the database operation request, and execute the execution plan corresponding to the database operation request when the class information matching the class identifier information exists in the local class loader. It can be seen that the above technical solution can be implemented in the case of executing the request processing process in the multiple end nodes of the distributed database, can reduce the network data transmission volume between the nodes when the distributed database processes the request, and further reduce the load and time delay of the request processing process of the distributed database, thereby solving the problems of high network data transmission volume, load and time delay when the existing distributed database processes the request, and improving the request processing efficiency of the distributed database.

[0137] The request processing apparatus of the distributed database can execute the request processing method of the distributed database executed by the client node provided by any embodiment of the present application, has the function modules and beneficial effects corresponding to the execution method. The technical details not described in detail in the present embodiment can be referred to the request processing method of the distributed database executed by the client node provided by any embodiment of the present application.

[0138] Embodiment five

[0139] Figure 8 A structural diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0140] As shown in Figure 8 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., connected to the at least one processor 11 in communication, where the memory stores computer programs executable by the at least one processor 11, and the processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or loaded into the random access memory (RAM) 13 from the storage unit 18. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0141] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0142] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the request processing method of the distributed database.

[0143] Optionally, the request processing method of the distributed database applied to the coordination node can include: receiving a database operation request submitted by the distributed database, and parsing the database operation request to obtain a request parsing result; determining a processing data volume of the database operation request according to the request parsing result; determining a target execution node of the database operation request according to the processing data volume of the database operation request; wherein the target execution node includes a client node, a coordination node or a worker node; and generating an execution plan corresponding to the database operation request, so that the target execution node executes the execution plan corresponding to the database operation request.

[0144] Optionally, the request processing method of the distributed database applied to the client node can include: receiving an execution plan corresponding to a database operation request generated by a coordination node; determining class identification information of the database operation request according to the execution plan corresponding to the database operation request; and executing the execution plan corresponding to the database operation request in a case where class information matched with the class identification information exists in a local class loader.

[0145] In some embodiments, the request processing method of the distributed database can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the request processing method of the distributed database described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the request processing method of the distributed database by any other suitable means, such as by means of firmware.

[0146] The various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0147] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, enables the functions / acts specified in the flowcharts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0148] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0149] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0150] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), blockchain network, and the Internet.

[0151] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0152] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present disclosure are achieved, and the present disclosure is not limited herein.

[0153] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A request processing method for a distributed database, characterized in that, Applied to coordinating nodes, including: Receive database operation requests submitted to a distributed database, parse the database operation requests, and obtain the request parsing results; The amount of data to be processed in the database operation request is determined based on the request parsing result. The target execution node for the database operation request is determined based on the amount of data to be processed in the database operation request; wherein, the target execution node includes a client node, a coordinating node, or a worker node; An execution plan corresponding to the database operation request is generated so that the target execution node executes the execution plan corresponding to the database operation request.

2. The method according to claim 1, characterized in that, Determining the amount of data to be processed for the database operation request based on the request parsing result includes: The amount of input parameters, output parameters, and intermediate data retrieved for the database operation request are determined based on the request parsing results. Calculate the total amount of data including the input data, the output data, and the intermediate fetched data.

3. The method according to claim 2, characterized in that, Determining the target execution node of the database operation request based on the amount of data to be processed in the database operation request includes: If the total amount of data in the database operation requests is less than or equal to a first set data amount threshold, the coordination node is determined as the target execution node. If the total amount of data in the database operation requests is greater than the first set data amount threshold, the amount of input parameter data is greater than the second set data amount threshold, and the amount of intermediate data retrieved is less than the third set data amount threshold, then the client node is determined as the target execution node. If the total amount of data in the database operation requests is greater than the first set data amount threshold, the amount of output parameter data is greater than the fourth set data amount threshold, and the amount of intermediate data pulled is less than the third set data amount threshold, then the client node is determined as the target execution node. If the total amount of data in the database operation requests is greater than the first set data amount threshold, the amount of input parameter data is less than or equal to the second set data amount threshold, the amount of output parameter data is less than or equal to the fourth set data amount threshold, and the amount of intermediate data pulled is greater than or equal to the third set data amount threshold, then the working node is determined as the target execution node.

4. A request processing method for a distributed database, characterized in that, Applied to client nodes, including: Receive the execution plan corresponding to the database operation request generated by the coordinating node; The class identifier information of the database operation request is determined based on the execution plan corresponding to the database operation request; If it is determined that a class matching the class identifier exists in the local class loader, the execution plan corresponding to the database operation request is executed.

5. The method according to claim 4, characterized in that, Also includes: If it is determined that there is no class information matching the class identifier information in the local class loader, but there is class information matching the class identifier information in the local class information cache, the class information matching the class identifier information stored in the local class information cache is loaded into the local class loader, and the execution plan corresponding to the database operation request is executed. If it is determined that no class information matching the class identifier information exists in either the local class loader or the local class information cache, the class information matching the class identifier information is requested from the coordinating node based on the class identifier information. The class information matching the class identifier information is loaded into the local class information cache, and the class information matching the class identifier information stored in the local class information cache is further loaded into the local class loader. The execution plan corresponding to the database operation request is then executed.

6. A request processing apparatus for a distributed database, characterized in that, Configured on the coordinating node, including: The database operation request parsing module is used to receive database operation requests submitted to the distributed database, parse the database operation requests, and obtain the request parsing result; The data volume determination module is used to determine the data volume of the database operation request based on the request parsing result. The target execution node determination module is used to determine the target execution node of the database operation request based on the amount of data to be processed in the database operation request; wherein, the target execution node includes a client node, a coordinating node, or a worker node; An execution plan generation module is used to generate an execution plan corresponding to the database operation request, so that the target execution node executes the execution plan corresponding to the database operation request.

7. A request processing apparatus for a distributed database, characterized in that, Configured on the client node, including: The execution plan receiving module is used to receive the execution plan corresponding to the database operation request generated by the coordinating node; The request class information determination module is used to determine the class information of the database operation request based on the execution plan corresponding to the database operation request; The execution plan execution module is used to execute the execution plan corresponding to the database operation request when it is determined that the class information of the database operation request exists in the local class loader.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that is executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the request processing method of the distributed database according to any one of claims 1-3, or to perform the request processing method of the distributed database according to any one of claims 4-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are configured to cause a processor to execute the request processing method of the distributed database according to any one of claims 1-3, or to perform the request processing method of the distributed database according to any one of claims 4-5.

10. A computer program product, characterized in that, It includes a computer program / instruction, wherein when the computer program / instruction is executed by a processor, it implements the request processing method of the distributed database as described in any one of claims 1-3, or performs the request processing method of the distributed database as described in any one of claims 4-5.

Images