A distributed database system and a method for issuing query execution commands to the system.

By adopting a three-tier modular architecture and hybrid distribution mechanism for distributed database systems, the problems of limited query execution optimization and high network interaction overhead in distributed databases are solved. This enables efficient and integrated query execution, adapts to the query characteristics of different business scenarios, and improves system scalability and maintainability.

CN123115207APending Publication Date: 2026-11-17CHINA UNICOM DIGITAL TECNOLOGY CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611636800.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-22
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies in distributed databases suffer from limited query execution optimization effects, high network interaction overhead in complex query scenarios, and an inability to achieve efficient and integrated execution across the entire process.

Method used

It adopts a three-layer modular architecture of client, coordinator node and data node, and realizes the integration and optimization of distributed execution plans through standardized data flow interaction, avoiding duplicate calculations between nodes. It adopts a mixed distribution mechanism of query statement and execution plan to accurately distribute and uniformly manage the execution process.

Benefits of technology

It improves the query execution efficiency of distributed databases, reduces network overhead, adapts to the query characteristics of different business scenarios, enhances the scalability and maintainability of the system, and ensures the low-latency access requirements of high-concurrency transaction queries.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115207A_ABST
    Figure CN123115207A_ABST
Patent Text Reader

Abstract

This invention discloses a distributed database system and a method for issuing query execution commands to the system. The distributed database system includes a client, a coordinator node, and a cluster of peer data nodes. The modules perform standardized communication and rely on internal pathways to exchange intermediate results from shards, enabling cross-shard operations. The method includes: parsing and extracting physical routing keys to locate the target node, rewriting table names, processing special functions, and then issuing the SQL; generating a distributed execution plan through a CBO optimizer, segmenting task fragments, serializing operator structures, issuing them as binary streams via RPC, and establishing a data exchange pipeline. This invention integrates the distributed execution plan into the unified execution of the distributed database, solving the performance problem of issuing large round-trip times between the coordinator and data nodes based on query statements.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of database data processing technology, specifically to a distributed database system and a method for issuing query execution instructions for the system. Background Technology

[0002] With the explosive growth of the Internet, the Internet of Things and big data businesses, the amount of enterprise data is increasing exponentially, and high-concurrency read and write requirements have become the norm.

[0003] Traditional standalone databases are limited by the physical hardware limits of a single server, such as CPU, memory, disk I / O, and network bandwidth. When faced with massive amounts of data, they encounter serious performance bottlenecks: limited storage capacity, degraded computing performance, and insufficient high-concurrency handling capacity. Therefore, in order to break through the single-machine bottleneck, database architecture has begun to evolve towards a Shared-Nothing distributed architecture, which achieves scale-out by horizontally splitting (Sharding) data and storing it on multiple independent physical nodes.

[0004] In distributed databases, to shield upper-layer applications from the complex underlying physical distribution, a common architecture is the separation of computation and storage or peer / master-slave collaboration. In this case, two core roles evolve within the system: first, the coordinator node, responsible for receiving raw SQL requests from clients, parsing the SQL, generating distributed execution plans, coordinating transactions, and summarizing results; second, the data nodes, responsible for managing the actual data shards and possessing local computational capabilities (such as local index lookup, filtering, and aggregation). Under this architecture, efficiently completing a query becomes the core challenge in distributed database design.

[0005] The technology for issuing query statements or execution plans was created to solve the following core problem: from "data movement-oriented" to "computation movement-oriented". In traditional cold backup or simple database sharding architecture, if it is necessary to associate data across nodes, it is usually necessary to extract the data of all nodes into memory or a central node for computation. However, this "data movement" method will generate huge network I / O overhead.

[0006] The consensus in modern distributed databases is that mobile computing is more economical than mobile data. The essence of query delivery technology is to push down the computing logic (operators) to the physical node where the data is located, making full use of the local I / O and local computing power of the data node to achieve data locality.

[0007] In the field of distributed database query execution optimization technology, current mainstream similar implementation solutions mainly revolve around two core directions: optimized generation of execution plans and unified rewriting of query statements, in order to improve the query execution efficiency and operational consistency of distributed databases. Among them: The first type of solution is a distributed execution plan generation solution based on data node instances. The core idea of ​​this solution is to combine the actual running status, storage and computing capabilities of each data node instance to generate a query execution plan that is adapted to the distributed architecture. Through algorithm optimization, a better overall execution logic is constructed. Its core optimization concept is to push the data computing logic down as much as possible, avoid the drawbacks of centralized computing in traditional middleware, reduce the centralized transmission of data to the database middleware, and maximize the use of the local computing and storage resources of each data node. The second type of solution is a unified SQL rewriting and computation migration optimization solution based on the coordinator node. This solution adopts the core architecture concept of "computation migration-oriented". It replaces the traditional data migration computation mode with an interaction mode of direct distribution of query statements between the coordinator node and the data nodes. At the same time, a standardized and centrally managed query statement rewriting rule system is built on the coordinator node. The coordinator node uniformly rewrites, optimizes and distributes the original query statements initiated by the client, thereby unifying the query execution logic of the distributed database and improving the overall running performance and data consistency. For example, the existing patent with patent publication number CN107798017A, "Method and System for Generating Execution Plan Information in Distributed Databases," discloses a distributed database execution plan information optimization generation technology. This technology can assist developers in optimizing the logic of query statements, fully explore the computing and storage resource potential of distributed databases, effectively avoid the problem of large amounts of data being transmitted over the network to the database middleware for computation, and significantly improve the overall execution efficiency of a single query statement, providing basic technical support for the optimization of execution plans on the data node side.

[0008] Alternatively, there is the existing patent CN121388050A, "Dynamic SQL Rewriting Method under Distributed Database," which discloses a dynamically iterative unified SQL rewriting technology. This technology relies on a coordinator node to centrally manage and dynamically update rewriting rules. It can comprehensively rewrite and optimize SQL statements corresponding to four mainstream execution plans: LightProxy, Stream, FQS, and RemoteQuery. It also features transparent compatibility, client-side insensitivity, and dynamic scalability. Changes to rewriting rules can be instantly synchronized to the coordinator's local memory, ensuring real-time effectiveness of the rewriting logic and effectively improving the performance, consistency, and adaptability of the distributed database.

[0009] While the aforementioned existing technologies can optimize distributed database query execution performance to some extent, they still have significant technical shortcomings in practical applications and end-to-end execution adaptation, failing to achieve efficient and integrated query execution optimization across the entire process. Specific deficiencies include: (1) For distributed execution plan generation technology based on data node instances, there is a problem of insufficient implementation. Conventional technical solutions can only provide reference for front-end developers to optimize query statements and realize the pre-optimization guidance of query statements, but they do not truly implement the optimized distributed execution plan. They cannot directly distribute the standardized optimal execution plan to each data node instance for execution. The overall optimization logic of the technology only stays at the statement reference level and fails to deeply integrate into the entire chain of query scheduling, distribution, and execution of distributed databases. It cannot realize the integrated closed-loop execution of the database, and the optimal configuration effect of data node computing resources is difficult to be truly realized, resulting in limited overall optimization effect. (2) For dynamic SQL rewriting technology based on coordinator nodes, there are problems such as poor adaptability to complex queries and high execution overhead. Although conventional technical solutions can realize unified rewriting and centralized management of SQL statements and adapt to various execution plan optimization scenarios, the node interaction mode is singular. The coordinator node and data node only support the basic query statement distribution method. After the distributed query statement arrives at the data node, the data node still needs to repeat the entire process of statement parsing and execution plan reconstruction, generating a large amount of redundant computing overhead. Especially for complex query scenarios with multiple conditions and dimensions, the technology often needs to rewrite and generate multiple query statements and distribute them in batches, which will greatly increase the number of network interaction round trips between the coordinator node and the data node, resulting in serious network overhead and scheduling delay, significantly reducing the overall query performance in complex query scenarios, and failing to meet the query execution requirements of high concurrency and complex business.

[0010] To address this, this application proposes a distributed database system and a method for issuing query execution instructions for the system, which can integrate the distributed execution plan into the unified execution of the distributed database and solve the performance technical problem of issuing large round-trip times between the coordinator and data nodes based on query statements. Summary of the Invention

[0011] The main objective of this invention is to provide a distributed database system and a method for issuing query execution instructions for the system, so as to solve the technical problems mentioned in the background art.

[0012] The present invention solves the above-mentioned technical problems by adopting the following technical solutions: A distributed database system, comprising: The client module serves as the query entry point on the business side, receiving user business query requests, generating raw SQL statements, and sending them one-way down to the coordinator node module. During the interaction, the client module only acts as the query initiator and does not participate in the optimization, routing, or execution scheduling of SQL statements. It has no distributed computing capabilities, only completes the output of business statements, and passes the raw SQL statements to the lower-level coordinator node for processing. The coordinator node module, as the central hub for distributed cluster scheduling, receives raw SQL statements from client modules and performs end-to-end statement optimization, routing determination, and branch distribution operations. The data node module consists of multiple peer data node instances in the cluster, and the nodes exchange intermediate results across nodes through data paths. Modules interact through standardized data flow to achieve cross-node collaboration. Nodes rely on data paths to transmit intermediate calculation results of shards, complete cross-shard association and aggregation operations, and finally send the query results back to the coordinator node for aggregation and return to the client.

[0013] Preferably, the coordinator node module is internally configured with: The SQL parsing submodule receives the raw SQL statement sent by the client module, performs syntax and lexical parsing, generates a structured abstract syntax tree, extracts core query metadata including query tables, filtering conditions, aggregation logic, and sharding fields, and outputs the structured query information metadata to the logic optimization submodule. The logic optimization submodule is used to perform distributed query logic layer optimization operations based on the metadata output by the SQL parsing submodule. These operations include pruning invalid associations, pushing down filter conditions, merging aggregation operations, and eliminating redundant projections. The optimized logical query tree is then output and synchronously sent to the routing decision submodule. The routing decision submodule is used to determine sharding routes for the logical query tree by combining sharding rules, storage distribution of each data node, and node load status. It has two types of branch processing logic: Branch 1: Trigger SQL reverse parsing and rewrite submodule; Branch 2: Trigger the issuance of the execution plan and execute the sub-module; The SQL reverse parsing and rewriting submodule (optimization branch) has a built-in unified management rewriting rule library. It is used to receive the logical query tree output by the routing decision, reverse parse the structured logical query tree into standardized SQL statements adapted to sharding, and complete the sharding statement rewriting, splitting, and filtering condition adaptation rewriting. After the rewriting is completed, the corresponding SQL statement is sent to the target data node. The execution plan delivery and execution submodule differs from existing technologies that only deliver SQL statements. This module physically decomposes the optimized logical query tree to generate a distributed physical execution plan tree containing multiple types of execution operators. It can encapsulate remote cross-node execution operators into dedicated control structures and supports serialization and lightweight delivery of control structures. Finally, it directly serializes the complete distributed physical execution plan generated by the routing decision and delivers the serialized physical execution plan to the corresponding data nodes. This eliminates the need for data nodes to repeatedly parse SQL statements and rebuild execution plans, directly delivering the data node engine for execution.

[0014] Preferably, each data node module has only one set of core execution modules. The core execution module is used to execute the processing logic of two input data streams, and can identify and parse the remote execution operator control structure issued by the coordinator node and complete the corresponding operation execution. (1) Receive the rewritten SQL statement issued by the coordinator and execute the logic: autonomously complete the parsing of the SQL statement and generate a local execution plan before calculation; (2) Receive the complete physical execution plan issued by the coordinator and execute the logic: directly load the execution plan, skip the redundant steps of SQL parsing and local execution plan generation, and finally complete the data reading, calculation and aggregation locally.

[0015] Preferably, the standardized data flow during the interaction between modules includes: (a) Upstream data stream, used by the client to transmit raw SQL statements unidirectionally to the coordinator.

[0016] (b) The coordinator’s internal serial processing flow is used for the coordinator to sequentially execute the parsing, logic optimization and routing decision operations of SQL statements.

[0017] (c) The data flow distributed by the coordinator branch is characterized as a data flow with two distribution paths based on the routing decision: Perform unparse rewriting on the SQL statement and send the rewritten SQL statement to the data stream of the data node; And generate a complete physical execution plan, and distribute the execution plan to the data nodes for execution.

[0018] (d) Internal data layer interaction flow, characterized as a data flow in which multiple data nodes rely on the internal data path between nodes to transmit intermediate data generated by sharding computation in both directions.

[0019] (e) Result feedback data stream, used by data nodes to push the calculation results to the coordinator after completing the calculation, and the coordinator to summarize the results from multiple parties and return the final query result to the client.

[0020] On the one hand, the present invention also discloses a method for issuing query execution instructions, which, based on any of the above-mentioned distributed database systems, implements the SQL statement issuance operation of the corresponding SQL execution SQL reverse parsing and rewriting submodule through a computer device, including: Step S1. After receiving the original SQL statement, the coordinator node module extracts the physical routing key (ShardingKey) through a set of parsers. Here, the physical routing key is the field / value used to locate the shard and determine which data node the data belongs to when the data is actually sharded and stored. Step S2. Based on the extracted physical routing key, address and determine the single or several target physical data nodes to be accessed; Step S3. SQL statement rewriting: If the original SQL statement contains a logical table name, the coordinator node module replaces it with the physical table name on the target physical data node; If there are functions that cannot be executed directly across nodes, then the corresponding stripping or rewriting operations are performed to rewrite them into corresponding data retrieval statements and send them to the corresponding data nodes for the corresponding function execution operations on the coordinator node. Step S4. Send the standard SQL string or the pipelined statement with parameters, after the rewriting operation performed in step S3, to the data node via a standard database protocol (such as the MySQL protocol).

[0021] Preferably, the specific operation process of step S1, which extracts the physical routing key through a set of resolvers, includes: Query the data table to obtain the distribution key column (e.g., column 1 of the table), the distribution algorithm (e.g., hash, modulo, etc.), and the list of data distribution nodes (e.g., distributed on backend data nodes A, B, and C) for the metadata set in all coordinator nodes. When the predicate retrieved by the query contains a distribution key filter, the data node for the actual data storage is accurately located based on the value and the distribution algorithm (e.g., the statement is SELECT). The expression `from table A where column 1 = value A` can be used to calculate the target data node as node A by taking the modulo of value A.

[0022] On the other hand, the present invention also discloses a method for issuing query execution instructions, based on any of the above-mentioned distributed database systems, using computer equipment to implement the issuance of corresponding execution plans and the issuance of physical execution plans for execution sub-modules, including: Preferably, step L1. The CBO optimizer (cost-based optimizer) of the coordinator node module is used to reconstruct and split the physical execution plan logic into distributed physical execution plans to generate a global, cross-node distributed physical execution plan tree. The generation step of the physical plan tree adopts the current mainstream database generation algorithm and corresponds to the remote execution algorithm. Its control structure is a simple encapsulation of the corresponding plan node and includes routing information (mainly the list of data nodes involved). Step L2. Divide the execution plan tree into multiple task segments that can run independently on a single node. The task segments are used to execute the corresponding encapsulated operation of a certain execution node in a certain execution plan tree. Each task segment contains multiple sets of execution operators. Therefore, for each remote execution operator, a control structure for generating the remote execution operator is encapsulated. Step L3. Serialization: The coordinator node module is then used to serialize the control structure containing the remote execution operator to generate a lightweight binary data stream; Step L4. Finally, the corresponding operator tree will be executed to distribute the corresponding task to the established data link channels of each data node.

[0023] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0024] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0025] As can be seen from the above technical solution, the present invention provides a distributed database system and a method for issuing query execution instructions for the system. Compared with the prior art, the present invention has the following advantages: 1. This invention integrates the distributed execution plan output by the distributed optimizer into the unified execution chain of the distributed database, making the distributed execution plan the core carrier for computational movement. This avoids repeatedly performing SQL parsing, logic optimization, SQL de-parsing, and execution plan generation operations between various nodes, thereby reducing redundant computational overhead across the entire chain and lowering system resource consumption.

[0026] 2. This invention employs a hybrid delivery mechanism of query statements and execution plans, relying on query statement classification and pre-filtering to achieve precise traffic splitting between the two delivery modes. This avoids the drawback of simply delivering based on rewritten SQL, which leads to an increase in the number of network round trips between the coordinator and data nodes.

[0027] 3. This invention enables the coordinator to flexibly issue decisions, directly issuing SQL for local processing by data nodes in simple high-concurrency point query scenarios, and issuing finely distributed physical operator trees to uniformly manage the execution process for complex cross-node analysis queries. Therefore, it can adapt to the different query characteristics of TP transactional business and AP analytical business, and ultimately achieve optimal system execution efficiency under mixed business loads.

[0028] 4. This invention reduces the computational and serialization pressure on the coordinator by reusing the built-in single-machine optimizer of the data node through SQL distribution, which can ensure the low-latency access requirements of high-concurrency transaction queries and stably support TP business scenarios. At the same time, it breaks the boundaries of the single-machine execution engine by distributing execution plans and relies on the global view of the coordinator for unified scheduling, which can realize large-scale distributed parallel computing.

[0029] 5. This invention adopts a three-layer modular architecture consisting of client, coordinator node, and data node. The modules are divided into independent functional units, and the modules interact with each other based on standardized data flow. This enables the complete connection of the entire business chain, from distributed pre-optimization and execution plan distribution to local engine execution, ultimately improving the maintainability and scalability of the system architecture.

[0030] It should be understood that the descriptions in this section are not intended to identify key or essential features of embodiments of the invention, nor are they intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Of course, implementing any product of the invention does not necessarily require achieving all of the advantages described above simultaneously. Attached Figure Description

[0031] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart illustrating the overall data processing flow of the present invention. Figure 2 This is a schematic diagram of the hybrid query statement issuance process according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the execution plan issuance process according to an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] For details in the embodiments, please refer to Figures 1 to 3 .

[0034] The distributed database system proposed in this embodiment of the invention has the following data processing flow: Figure 1 As shown, the system specifically includes: (i) The client module serves as the entry point for business-side queries, receiving user business query requests, generating raw SQL statements, and sending them one-way down to the coordinator node. During the interaction, the client module only acts as the query initiator, does not participate in SQL optimization, routing, or execution scheduling, has no distributed computing capabilities, and only completes the output of business statements, passing the raw SQL to the lower-level coordinator node for processing.

[0035] (ii) The coordinator node module is the central hub for distributed cluster scheduling. It receives the client's original SQL and performs end-to-end statement optimization, routing determination, and branch distribution. Internally, it is divided into 5 serial / branch sub-modules: (1) SQL parsing submodule, which is used to receive the original SQL statement sent by the client, complete the syntax and lexical parsing, generate a structured abstract syntax tree, extract the core query metadata such as query table, filtering conditions, aggregation logic, and sharding fields, and output the structured query information to the logic optimization submodule.

[0036] (2) The logic optimization submodule is used to perform distributed query logic layer optimization based on the query metadata output by SQL parsing: pruning invalid associations, pushing down filtering conditions, merging aggregation operations, eliminating redundant projections, and outputting the optimized logical query tree, which is then synchronously sent to the routing decision submodule.

[0037] (3) The routing decision submodule is used to determine the sharding route of the logical query tree by combining the sharding rules, the storage distribution of each data node, and the node load status. There are two types of branch processing logic: Branch 1: Trigger SQL reverse parsing and rewrite submodule; Branch 2: Trigger the issuance of the execution plan and the execution of the sub-module.

[0038] (4) SQL reverse parsing and rewriting submodule (optimization branch) has a built-in unified management rewriting rule library, which is used to receive the logical query tree output by the routing decision, reverse parse the structured query tree into standardized SQL that adapts to the sharding, complete the sharding statement rewriting, splitting, and filtering condition adaptation rewriting, and send the corresponding SQL to the target data node after the rewriting is completed.

[0039] (5) The execution plan distribution and execution sub-module differs from existing technologies that only distribute SQL statements. This module directly serializes the complete distributed physical execution plan generated by the routing decision and distributes it to the corresponding data nodes. The data nodes do not need to repeatedly parse the SQL and rebuild the execution plan. The data nodes are directly delivered to the data node engine for execution.

[0040] (iii) Data node module: The cluster contains multiple peer data node instances, and the nodes complete the cross-node intermediate result interaction through data paths. Each node has only a core execution unit.

[0041] The core execution unit includes: (1) Database execution engine submodule, used to execute the processing logic of two input data streams: (a) Receive the rewritten SQL from the coordinator: autonomously complete SQL parsing, generate a local execution plan, and then perform the operation; (b) Receive the complete physical execution plan issued by the coordinator: directly load the execution plan, skip the redundant steps of SQL parsing and plan generation, and complete data reading, calculation and aggregation locally; Ultimately, cross-node collaboration is achieved: nodes rely on data paths to transmit intermediate calculation results of shards, complete cross-shard association and aggregation operations, and finally send the query results back to the coordinator node for aggregation before returning them to the client.

[0042] At this point, the data flow interaction relationship between modules is as follows: (a) Upstream data flow: Client → Coordinator (raw SQL one-way transmission); (b) Serial flow within the coordinator: SQL parsing → logical optimization → routing decision; (c) Coordinator branch flow: Routing decision → SQL de-parsing and rewriting → Data node (deploying rewritten SQL); Routing decision → Execution plan issuance and execution → Data node (issuance of complete physical execution plan); (d) Internal data layer flow: Multiple data nodes transmit fragmented intermediate data bidirectionally through the internal data path between nodes; (e) Result feedback stream: Data node calculation results → Coordinator summary → Client returns final query results.

[0043] By adopting a three-layer modular architecture consisting of client, coordinator node, and data node, with independent functional units within each module and standardized data flow interaction between modules, the entire business chain of distributed pre-optimization, execution plan distribution, and local engine execution can be fully connected, ultimately improving the maintainability and scalability of the system architecture.

[0044] In summary, this system integrates the distributed execution plan output by the distributed optimizer into the unified execution chain of the distributed database. This makes the distributed execution plan the core carrier for computational movement, avoiding the repeated SQL parsing, logic optimization, SQL de-parsing, and execution plan generation operations between various nodes. This reduces redundant computational overhead across the entire chain and lowers system resource consumption.

[0045] Furthermore, in the specific implementation process, it can also give the coordinator the ability to flexibly issue decisions. It can directly issue SQL for simple high-concurrency point query scenarios and let the data nodes process it locally. It can also issue a fine-grained distributed physical operator tree to uniformly manage the execution process for complex cross-node analysis queries. Therefore, it can adapt to the different query characteristics of TP transactional business and AP analytical business, and ultimately achieve the optimal system execution efficiency under mixed business load.

[0046] Meanwhile, the system also reduces the computation and serialization pressure on the coordinator by reusing the built-in single-machine optimizer of the data node through SQL distribution, which can ensure the low-latency access requirements of high-concurrency transaction queries and stably support TP business scenarios. At this time, it also breaks the boundaries of the single-machine execution engine by distributing execution plans and relies on the coordinator's global view for unified scheduling, which can realize large-scale distributed parallel computing.

[0047] On the other hand, this invention also discloses a method for issuing query execution instructions, which includes the following core operation paths and supports technical details of issuing mixed query statements and execution plans: Path 1: Query statement is issued; the specific data processing flow is as follows: Figure 2 As shown.

[0048] Applicable scenarios: DDL statements, single-shard point queries, single key-value writes (Insert / Update / Delete), and simple unrelated range queries.

[0049] The execution steps of the technical implementation: (1) The coordinator receives the original SQL and extracts the physical routing key (ShardingKey) through the parser.

[0050] (2) Calculate the unique or few physical data nodes where the data is located.

[0051] (3) SQL rewriting: If the original SQL contains logical table names, the coordinator replaces them with the physical table names on the target physical data nodes; if it contains functions that cannot be directly executed across nodes, it is stripped or rewritten.

[0052] (4) Send the rewritten standard SQL string (or the Pipelined statement with parameters) to the data node through a standard database protocol (such as the MySQL protocol).

[0053] Path 2: Physical execution plan / operator issuance, specific data processing flow as follows Figure 3 As shown.

[0054] Applicable scenarios: Distributed analytical scenarios (MPP scenarios) such as cross-node join, large-scale data aggregation (GroupBy), and complex subqueries.

[0055] The execution steps of the technical implementation: (1) The coordinator’s CBO (cost-based optimizer) breaks down the logical plan and generates a global, cross-node distributed physical execution plan tree.

[0056] (2) The execution plan tree is divided into multiple task segments that can run independently on a single node.

[0057] (3) Serialization: The coordinator serializes these control structures containing operators executed remotely into lightweight binary streams.

[0058] Finally, the task distribution is executed: the corresponding operator tree (a physical execution plan tree organized in a tree structure by various execution operators) is distributed to the RPC listening port of each data node.

[0059] In summary, this method employs a hybrid delivery mechanism of query statements and execution plans, relying on query statement classification and pre-filtering to achieve precise traffic splitting between the two delivery modes. This avoids the drawback of simply delivering based on rewritten SQL, which leads to an increase in the number of network round trips between the coordinator and data nodes.

[0060] Furthermore, it should be noted that in practical applications, the coordinator can adaptively choose to issue standardized SQL query statements or serialized distributed physical execution plans / operators based on query complexity, number of shards, and query cost.

[0061] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0062] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0063] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the query execution instruction issuing method of any of the distributed database systems described above.

[0064] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0065] This application also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus. Memory, used to store computer programs; When the processor executes a program stored in memory, it implements the query execution instruction issuance method of the aforementioned distributed database system.

[0066] The communication bus mentioned in the above-mentioned electronic devices can be a standard bus for interconnecting peripheral components or an extended industrial standard structure bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0067] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0068] The memory may include random access memory or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0069] The processors mentioned above can be general-purpose processors, including central processing units, network processors, etc.; they can also be digital signal processors, application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0070] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0072] Furthermore, it should be noted that if any directional indication (such as up, down, left, right, front, back, etc.) is involved in the embodiments of the present invention, the directional indication is only used to explain the relative positional relationship and movement of each component in a specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0073] Furthermore, those skilled in the art should understand that in the actual use of the embodiments of this application, there may be preset thresholds used as the basis for judging the corresponding technical solutions. These thresholds are conventional technical means commonly used in the field to implement functions such as state judgment, condition recognition, and control logic switching. The specific values, setting basis, value selection methods, determination methods, and adjustment rules of the thresholds involved in this technical solution are all conventional technical choices that can be reasonably determined by those skilled in the art based on conventional technical factors such as actual application scenarios, system working states, characteristics of the detection object, hardware performance parameters, and functional requirements, through conventional experiments, calibrations, and debugging. The specific setting and adjustment of the aforementioned thresholds will not cause this technical solution to be unimplementable as a whole, nor will it affect the realization of the core concept and the achievement of the technical effects of this technical solution.

[0074] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, in the embodiments of this invention, "multiple" refers to two or more. Moreover, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

Claims

1. A distributed database system, characterized in that, include: The client module serves as the entry point for business-side queries, receiving user business query requests, generating raw SQL statements, and sending them one-way down to the coordinator node module. The coordinator node module is used to receive the raw SQL statements issued by the client module and complete the entire chain of statement optimization, routing determination, and branch distribution operations. The data node module consists of multiple peer data nodes in the cluster, and the nodes exchange intermediate results across nodes through data paths. Modules interact through standardized data flow to achieve cross-node collaboration. Nodes rely on data paths to transmit intermediate calculation results of shards, complete cross-shard association and aggregation operations, and finally send the query results back to the coordinator node for aggregation and return to the client.

2. The distributed database system as described in claim 1, characterized in that, The coordinator node module is internally configured with: The SQL parsing submodule is used to receive raw SQL statements from the client module, perform syntax and lexical parsing, generate a structured abstract syntax tree, extract metadata including query tables, filtering conditions, aggregation logic, and sharding fields, and output structured query information metadata. The logic optimization submodule is used to perform distributed query logic layer optimization operations based on the metadata output by the SQL parsing submodule, including pruning invalid associations, pushing down filtering conditions, merging aggregation operations, and eliminating redundant projections. After that, it outputs the optimized logical query tree and sends it to the routing decision submodule synchronously. The routing decision submodule is used to make sharding routing decisions on the logical query tree by combining sharding rules, storage distribution of each data node, and node load status. The SQL reverse parsing and rewriting submodule has a built-in unified management rewriting rule library. It is used to receive the logical query tree output by the routing decision, reverse parse the structured logical query tree into standardized SQL statements adapted to sharding, and complete the sharding statement rewriting, splitting, and filtering condition adaptation rewriting. After the rewriting is completed, the corresponding SQL statement is sent to the target data node. The execution plan distribution and execution submodule directly serializes the complete distributed physical execution plan generated by the routing decision and distributes it to the corresponding data nodes.

3. The distributed database system as described in claim 2, characterized in that, Each data node module contains only one set of core execution modules, which are used to execute the processing logic of two input data streams: (1) Receive the rewritten SQL statement issued by the coordinator and execute the logic: autonomously complete the parsing of the SQL statement and generate a local execution plan before calculation; (2) Receive the complete physical execution plan issued by the coordinator and execute the logic: directly load the execution plan, skip the SQL parsing and local execution plan generation steps, and finally complete the data reading, calculation and aggregation locally.

4. A method for issuing query execution instructions, characterized in that, For use in a distributed database system based on any one of claims 1-3, implementing SQL statement distribution operations in the SQL reverse parsing and rewriting submodule, including: Step S1. After receiving the original SQL statement, the coordinator node module extracts the physical routing key through a set of parsers; Step S2. Based on the extracted physical routing key, determine the target physical data node to be accessed; Step S3. If the original SQL statement contains a logical table name, the coordinator node module replaces it with the physical table name on the target physical data node; If there are functions that cannot be executed directly across nodes, then the corresponding stripping or rewriting operations should be performed; Step S4. Send the standard SQL string or piped function statement with parameters after the operation in step S3 to the data node through the standard database protocol.

5. The query execution instruction issuance method as described in claim 4, characterized in that, The specific operation process of extracting the physical routing key through a set of resolvers in step S1 includes: Query the data table to obtain the distribution key column, distribution algorithm, and list of nodes used to determine the distribution, for the metadata set in all coordinator nodes that stores data distribution information; When the predicate retrieved by the query contains a distribution key filter, the data node for the actual data storage is located based on the value and the distribution algorithm.

6. A method for issuing query execution instructions, characterized in that, For use in a distributed database system based on any one of claims 1-3, implementing execution plan distribution and physical execution plan distribution operations for execution submodules, including: Step L1. Use the CBO optimizer of the coordinator node module to construct and split the physical execution plan logic into distributed physical execution plans to generate a distributed physical execution plan tree across nodes; Step L2. Divide the execution plan tree into multiple task segments that can run independently on a single node. The task segments are used to execute the corresponding encapsulation operations of the node. Each task segment contains multiple sets of execution operators. For each remote execution operator, encapsulate and generate the control structure of the remote execution operator. Step L3. Then use the coordinator node module to serialize the control structure containing the remote execution operator to generate a lightweight binary data stream; Step L4. Finally, the corresponding operator tree will be executed to distribute the corresponding task to the established data link channels of each data node.

Citation Information

Patent Citations

  • Method and system of execution plan information generation in distributed database

    CN107798017A

  • Dynamic SQL (Structured Query Language) rewriting method under distributed database

    CN121388050A