Data processing method and device, electronic equipment and storage medium

Abstract

This application discloses a data processing method, apparatus, electronic device, and storage medium, comprising: obtaining a logical plan of a structured query statement, wherein the logical plan reflects the logical operation steps of the structured query statement through a tree structure; identifying a target sub-plan based on the tree structure of the logical plan; and if the child nodes of a connection node are not aggregation nodes, and the target sub-plan meets a preset aggregation condition, pushing some aggregation operations down from the aggregation node to before the connection node, so as to perform some aggregation operations on the data tables connected by the connection node before executing the connection operation. If the child nodes of the connection node of the target sub-plan are not aggregation nodes, and the target sub-plan meets the preset aggregation condition, some aggregation operations can be pushed down to before the connection node, thereby achieving: performing some aggregation operations on the data packets before the connection operation, significantly reducing the amount of data processed by the connection node, thereby avoiding the processing of redundant information and reducing the computational burden.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In existing technologies, when performing join operations in a distributed environment, a common practice is to distribute data with the same key to the same nodes through shuffle to facilitate matching and joining. Shuffle refers to the data redistribution process performed by a distributed system during certain logical operations (such as Join, Group By, Sort, etc.) to centralize data with the same key onto the same computing node for processing.

[0003] However, during the shuffle process, data needs to be redistributed to different nodes based on the join key, resulting in a large amount of network transmission. Furthermore, during network transmission, duplicate or redundant information from the original data is also processed, increasing the computational burden. Summary of the Invention

[0004] This application provides a data processing method, apparatus, electronic device, and storage medium that can improve the problem of increased computational burden in the prior art.

[0005] This application provides a data processing method, the method comprising: obtaining a logical plan of a structured query statement, the logical plan reflecting the logical operation steps of the structured query statement through a tree structure; identifying a target sub-plan based on the tree structure of the logical plan, wherein the target sub-plan includes an aggregation node and a connection node, and the connection node is a subordinate node of the aggregation node, the aggregation node being used to perform partial aggregation operations and a final aggregation operation; if the child node of the connection node is not an aggregation node, and the target sub-plan satisfies a preset aggregation condition, pushing the partial aggregation operation down from the aggregation node to before the connection node, so as to perform the partial aggregation operation on the data table connected by the connection node before performing the connection operation.

[0006] This application provides a data processing apparatus, the apparatus comprising:

[0007] The logical plan acquisition unit is used to acquire the logical plan of the structured query statement. The logical plan reflects the logical operation steps of the structured query statement through a tree structure.

[0008] The sub-plan identification unit is used to identify target sub-plans based on the tree structure of the logical plan, wherein the target sub-plan includes aggregation nodes and connection nodes, and the connection nodes are the subordinate nodes of the aggregation nodes. The aggregation nodes are used to perform partial aggregation operations and final aggregation operations.

[0009] The operation pushdown unit is used to push down the partial aggregation operation from the aggregation node to before the connection node if the child node of the connection node is not an aggregation node and the target sub-plan meets the preset aggregation conditions, so as to perform the partial aggregation operation on the data table connected by the connection node before the connection operation is executed.

[0010] In one implementation, the preset aggregation condition includes the type of the aggregation node; correspondingly, the operation pushdown unit is specifically used to: if the child node of the connection node is not an aggregation node, and the type of the aggregation node is responsible for grouping data and does not include aggregation functions, then push down the partial aggregation operation from the aggregation node to before the connection node.

[0011] In one implementation, the preset aggregation condition further includes a broadcast operation state; correspondingly, the operation pushdown unit is specifically used for:

[0012] If the child node of the connection node is not an aggregation node, and the type of the aggregation node is responsible for grouping data and does not include aggregation functions, then the broadcast operation status is determined based on the data status of the data table connected by the connection node. The broadcast operation status includes suitable for broadcasting and unsuitable for broadcasting.

[0013] If the broadcast operation status is unsuitable for broadcasting, the partial aggregation operation is pushed down from the aggregation node to before the connection node.

[0014] In one implementation, operating the pushdown unit includes:

[0015] A node generation subunit is used to generate partial aggregation nodes based on partial aggregation operations of the aggregation nodes;

[0016] A node insertion subunit is used to insert the partial aggregation node between the connecting node and the child node of the connecting node;

[0017] The operation removes the subunit, which is used to remove part of the aggregation operation of the aggregation node.

[0018] In one embodiment, the device further includes:

[0019] The data table acquisition unit is used to acquire the data table suitable for broadcasting when the broadcast operation status is suitable for broadcasting;

[0020] The data sending unit is used to send the data from the data table suitable for broadcast to the worker nodes participating in the computation in the distributed system.

[0021] In one embodiment, operating the push-down unit further includes:

[0022] The table data volume acquisition subunit is used to acquire the data volume of the data table connected by the connection node.

[0023] The data volume comparison subunit is used to compare the data volume of the data table with the data volume threshold.

[0024] The suitable broadcast determination subunit is used to determine that the target data table is a suitable broadcast data table when there is a target data table with a data volume less than the data volume threshold.

[0025] In one implementation, broadcasting a designated subunit includes:

[0026] The distribution state sub-unit is used to determine the data distribution state of the target data table when there is a target data table with a data volume less than the data volume threshold.

[0027] The broadcast determination sub-unit is used to determine that the target data table is a suitable data table for broadcasting when the data distribution state is uniform.

[0028] In one implementation, the sub-plan identification unit includes:

[0029] The structure traversal subunit is used to traverse the tree structure of the logical plan to obtain multiple aggregate nodes;

[0030] The node type acquisition subunit is used to acquire the node type of the lower-level nodes of each aggregate node.

[0031] The sub-plan determination sub-unit is used to determine the sub-plan from the aggregation node to the lower-level node of the connection type when there is an aggregation node and the node type of the lower-level node of the aggregation node is connection type.

[0032] In one embodiment, the device further includes:

[0033] The data volume acquisition unit is used to acquire the data volume of the partial aggregation result connected by the connection node, wherein the partial aggregation result is the processing result obtained after the data table connected by the connection node undergoes a partial aggregation operation;

[0034] The comparison unit is used to compare the data volume of the partial aggregation result with the data volume threshold;

[0035] A broadcast unit is used to send data of the partial aggregation results whose data volume is less than the data volume threshold to the worker nodes participating in the computation in the distributed system if the data volume of the partial aggregation results is less than the data volume threshold.

[0036] This application also provides an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute steps in any of the data processing methods provided in this application.

[0037] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the data processing methods provided in this application.

[0038] This application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps in any of the data processing methods provided in this application.

[0039] In the data processing method provided in this application embodiment, a logical plan for a structured query statement is obtained. Based on the tree structure of the logical plan, a target sub-plan is identified. The target sub-plan includes aggregation nodes and connection nodes, where the connection nodes are subordinate nodes of the aggregation nodes. When the child nodes of the connection nodes are not aggregation nodes, and the aforementioned target sub-plans satisfy preset aggregation conditions, some aggregation operations are pushed down from the aggregation nodes to before the connection nodes, so that some aggregation operations are performed on the data tables connected to the connection nodes before the connection operations are executed.

[0040] In this embodiment, a logical plan for a structured query statement can be obtained. Based on the tree structure of the logical plan, a target sub-plan, including aggregation nodes and join nodes, is identified, where the join nodes are subordinate nodes of the aggregation nodes. If the child nodes of the join nodes in the target sub-plan are not aggregation nodes, and the target sub-plan meets preset aggregation conditions, some aggregation operations can be pushed down before the join nodes. This achieves the following: first, perform partial aggregation operations on the data table to obtain partial aggregation results; then, perform join operations on the partial aggregation results; and finally, perform the final aggregation operation. Performing partial aggregation operations on the data packets before the join operation can significantly reduce the amount of data processed by the join nodes, thereby avoiding the processing of redundant information and reducing the computational burden. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1a This is a schematic diagram illustrating an application scenario of the data processing method provided in this application;

[0043] Figure 1b This is a flowchart illustrating the data processing method provided in an embodiment of this application;

[0044] Figure 1c (1) shows a schematic block diagram of the original logic plan;

[0045] Figure 1c (2) shows a schematic block diagram of the optimized logic plan;

[0046] Figure 1d A schematic flowchart illustrating the SQL query statement is shown.

[0047] Figure 2 This is a flowchart illustrating a data processing method provided in another specific embodiment of this application;

[0048] Figure 3 This is a schematic diagram of a data processing apparatus provided in one embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] This application provides a data processing method, apparatus, electronic device, and storage medium.

[0052] Specifically, the data processing device can be integrated into an electronic device, such as a terminal or server. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal and server can be directly or indirectly connected via wired or wireless communication; this application does not impose any restrictions on this connection.

[0053] In some embodiments, the data processing apparatus may also be integrated into multiple electronic devices, such as multiple servers, with the data processing method of this application being implemented by the multiple servers.

[0054] In some embodiments, the terminal can also be used as a server to implement some or all of the functions of a server.

[0055] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0056] Please see details Figure 1a The method provided in this application embodiment may include: obtaining a logical plan of a structured query statement, wherein the logical plan reflects the logical operation steps of the structured query statement through a tree structure; identifying a target sub-plan based on the tree structure of the logical plan, wherein the target sub-plan includes an aggregation node and a connection node, and the connection node is a subordinate node of the aggregation node, the aggregation node is used to perform partial aggregation operations and final aggregation operations; if the child node of the connection node is not an aggregation node, and the target sub-plan satisfies a preset aggregation condition, pushing the partial aggregation operation down from the aggregation node to before the connection node, so as to perform the partial aggregation operation on the data table connected by the connection node before performing the connection operation.

[0057] The above method can obtain the logical plan of the structured query statement and, based on the tree structure of the logical plan, identify the target sub-plan, which includes aggregation nodes and join nodes, and where the join nodes are subordinate nodes of the aggregation nodes. If the child nodes of the join nodes of the target sub-plan are not aggregation nodes, and the target sub-plan meets the preset aggregation conditions, some aggregation operations can be pushed down before the join nodes. This achieves the following: first, perform partial aggregation operations on the data table to obtain partial aggregation results; then, perform join operations on the partial aggregation results; and finally, perform the final aggregation operation. Performing partial aggregation operations on the data packets before the join operation can significantly reduce the amount of data processed by the join nodes, thereby avoiding the processing of redundant information and reducing the computational burden.

[0058] This application relates to the field of big data processing, particularly query optimization techniques in distributed computing engines. Specifically, it presents a high-performance optimization scheme in the Spark SQL environment based on Join operations and Cost-Based Optimization (CBO) to push down partial aggregation operators. CBO is a method for optimizing query execution plans. It estimates the cost of different execution plans by analyzing query statistics (such as table size, data distribution, and column cardinality) to select the optimal execution path.

[0059] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent should be explicitly obtained, and the collection, use and processing of related data should comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0060] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0061] In this embodiment, a data processing method is provided. For example... Figure 1b As shown, the specific process of this method may include the following steps 110 to 130:

[0062] 110. Obtain the logical plan of the structured query statement, wherein the logical plan reflects the logical operation steps of the structured query statement through a tree structure.

[0063] Structured Query Language (SQL) is a programming language specifically designed for managing and manipulating relational databases. As one of the most important and widely used languages ​​in the database field, SQL provides a standardized set of methods that enable users to interact with databases efficiently. Through SQL, users can perform a variety of complex operations, including but not limited to querying data, inserting new records, updating existing records, deleting records, and creating or modifying the database structure. These capabilities make SQL an indispensable tool in data management, data analysis, and system development.

[0064] SQL, as the standard language for interacting with databases, is widely used in various Database Management Systems (DBMS), such as MySQL, PostgreSQL, Oracle Database, Microsoft SQL Server, and SQLite. While these DBMSs may differ in their underlying implementations, they all support SQL as their core query language, ensuring high compatibility and portability. Furthermore, with the development of big data and cloud computing technologies, the application scope of SQL has further expanded. Many modern data processing frameworks (such as Apache Hive and Spark SQL) also support SQL syntax, enabling users to leverage the familiar SQL language to process large-scale datasets stored in distributed environments.

[0065] A logical plan is an abstract representation of an SQL query, providing a high-dimensional perspective on the query's semantics and operational flow. This representation focuses on expressing the user's intent and the logical steps of data processing, without delving into the specific details of how these steps are actually executed in the underlying database system. A logical plan is a mapping or transformation of the SQL query's semantics. This means that a logical plan accurately reflects the user's query intent, such as which tables to retrieve data from, which filtering conditions to apply, and which aggregation operations to perform. In this way, the logical plan ensures a true representation of the user's query intent.

[0066] The logical plan focuses on defining the logical structure of the query, including but not limited to the selection of data sources (e.g., tables and views), the data join methods (e.g., inner join, left join), the application of filtering conditions (WHERE clause), grouping and sorting rules (GROUP BY and ORDER BY), and aggregation operations (e.g., SUM, COUNT). While the logical plan details these operations, it does not specify which algorithm or technique should be used to perform them. For example, the logical plan might indicate that a table join operation is needed, but it will not explicitly specify whether the table join operation should use a nested loop join or a hash join.

[0067] Logical plans are typically presented in a tree structure, where each node represents a logical operation (such as scan, filter, aggregate, join, etc.), and edges indicate the direction of data flow. The root node is usually the final output, while the leaf nodes are the data sources (such as tables or views).

[0068] 120. Based on the tree structure of the logical plan, identify the target sub-plan, wherein the target sub-plan includes an aggregation node and a connection node, and the connection node is a subordinate node of the aggregation node, and the aggregation node is used to perform partial aggregation operations and final aggregation operations.

[0069] Aggregate nodes are used to perform grouping and aggregation operations on data. Common aggregation operations include SUM, COUNT, and AVG, which are used to calculate the summary value in each group. Specifically, aggregation operations are usually performed in two stages: partial aggregation and final aggregation. This staged design aims to optimize performance, reduce data transfer volume, and improve the overall efficiency of the system.

[0070] Partial aggregation refers to a local aggregation step that occurs on different nodes or partitions of data storage during distributed query execution. Since data is typically distributed across multiple nodes or partitions, each node first performs preliminary aggregation calculations on its locally stored data to reduce the amount of data that needs to be transmitted over the network. For example, in a distributed system, if a query requires calculating the sum of all data, each node will first calculate the partial sum of its local data and then pass these partial sums as intermediate results to the next step. In this way, partial aggregation can significantly reduce network bandwidth consumption and speed up subsequent computations.

[0071] Final aggregation refers to the process of globally summarizing the local aggregation results generated by each node or partition after all partial aggregations have been completed. The task of this stage is to further merge all intermediate results to obtain the global aggregation result for the entire dataset. For example, in the above example of calculating the sum, the final aggregation stage will add the partial sums of each node to obtain the sum of the entire dataset. Final aggregation is usually performed by a central node or coordinator to ensure that all local results are correctly included in the calculation.

[0072] A join node represents a connection operation between two datasets. Common join types include inner join, left join, and right join. Specifically, datasets can be represented in the form of data tables.

[0073] A join node is a child node of an aggregate node, meaning its level is lower than that of the aggregate node. In one implementation, the join node can be a child node of the aggregate node, and the target sub-plan can take the form of Aggregate => Join. Aggregate => Join means that data is first joined through the Join node. Then, the results of the join are passed to the Aggregate node for grouping and aggregation operations.

[0074] In another implementation, an intermediate node can exist between the connecting node and the aggregation node; for example, the connecting node is a child node of a child node of the aggregation node. Let's assume the intermediate node is a project node; then the target sub-plan can take the form Aggregate => Project => Join. Aggregate => Project => Join means that: data is first joined through the Join node; then the joined result is passed to the Project node for column selection or calculation. Finally, the result from the Project node is passed to the Aggregate node for grouping and aggregation operations.

[0075] In this context, the Project node refers to the logical operation responsible for selecting or calculating columns on the data. Simply put, the Project node determines which columns the final output data should contain, and whether these columns need to undergo any calculations or transformations.

[0076] It should be understood that the target sub-plan includes aggregation nodes and connecting nodes, and the connecting nodes are subordinate nodes of the aggregation nodes. Whether there are intermediate nodes between the connecting nodes and the aggregation nodes, or how many intermediate nodes exist, should not be construed as a limitation of this application.

[0077] Optionally, in one embodiment, step 120 may specifically include the following steps 121 to 123:

[0078] 121. Traverse the tree structure of the logical plan to obtain multiple aggregation nodes.

[0079] The tree structure of the logical plan can be traversed to find all aggregation nodes. Traversal methods can be implemented using algorithms such as Depth-First Search (DFS) or Breadth-First Search (BFS) to find all nodes with aggregation operations, such as the application points of aggregation functions like SUM, COUNT, and AVG.

[0080] 122. For each of the aggregate nodes, obtain the node type of the lower-level nodes of the aggregate node.

[0081] Optionally, you can examine all child nodes (including direct and indirect children) of each aggregation node and obtain their node types. This step is to determine whether there are nodes of the join type below the aggregation node, and is not limited to direct children.

[0082] Specifically, the node types of the lower-level nodes of an aggregate node can be obtained through recursive checking: starting from the aggregate node, recursively check all its child nodes and their child nodes down the hierarchy until the leaf nodes are reached. For each aggregate node, recursively visit all its child nodes and record their types. By recursively checking all the lower-level nodes of the aggregate node, potential target sub-plans can be identified more comprehensively, providing a more accurate basis for query optimization.

[0083] 123. If there exists an aggregation node and the node type of the subordinate node of the aggregation node is connection type, then the subordinate node of the aggregation node to the connection type is determined as the target sub-plan.

[0084] In step 123, it may be necessary to determine, based on the results of the first two steps, whether there exists an aggregation node whose subordinate nodes contain a join type. If such a case exists, the portion from the aggregation node to the join node is defined as the target sub-plan.

[0085] For example, let's assume the tree structure of the logical plan is as follows:

[0086]

[0087] In the example above, there are two aggregation nodes: the first aggregation node performs a SUM operation on the result of joining table1 and table2; the second aggregation node performs a COUNT operation on the data in table3.

[0088] For the first aggregation node:

[0089]

[0090] The direct child node of the aggregation node is a Project node. The direct child node of the Project node is a Join node. Therefore, it can be determined that the subordinate nodes of this first aggregation node include Project and Join.

[0091] Because there is a join node, the part from the aggregation node to the join node constitutes the target sub-plan.

[0092] For the second aggregation node:

[0093] Aggregate(GROUP BY t3.key,COUNT(t3.id))

[0094] └──Scan(table3)

[0095] The direct child node of the aggregation node is a Scan node. Therefore, it can be determined that the subordinate nodes of the second aggregation node only include Scan nodes and not connection nodes. Therefore, this part will not be considered part of the target sub-plan.

[0096] In the above implementation, by traversing and analyzing the tree structure of the logical plan, specific target sub-plans can be identified, specifically: structures where aggregation nodes contain join types in their lower-level nodes. This method not only helps in understanding the logical flow of query execution but also provides a basis for query optimization, helping the optimizer select a more efficient execution path. Specifically, identifying the pattern "aggregation => (intermediate node) => join" can significantly improve query performance.

[0097] 130. If the child node of the connection node is not an aggregation node, and the target sub-plan satisfies the preset aggregation condition, the partial aggregation operation is pushed down from the aggregation node to before the connection node, so that the partial aggregation operation is performed on the data table connected by the connection node before the connection operation is executed.

[0098] Preset aggregation conditions are pre-defined conditions used to determine whether to push some aggregation operations down from the aggregation node to the connection node. The specific details of the preset aggregation conditions will be described in detail below.

[0099] Pushdown refers to the technique of pushing query operations or computations down to the data source layer as much as possible to reduce data transfer volume and improve query performance. By pushing operations to the data source, filtering, aggregation, or computation can be performed at the data source, so that only the filtered, aggregated, or computed data is transmitted to the Spark cluster for further processing.

[0100] If the child node of a connecting node is not an aggregator node, further determine whether the target sub-plan meets the preset aggregation conditions. If the preset aggregation conditions are met, some aggregation operations can be pushed down from the aggregator node to before the connecting node. Pushing some aggregation operations down from the aggregator node to before the connecting node is an important optimization strategy. This optimization strategy can reduce the amount of data involved in the connection by performing local aggregation of data in advance, thereby reducing network transmission overhead and computing resource consumption.

[0101] Optionally, in one implementation, step 130, "before pushing the partial aggregation operation down from the aggregation node to the connection node," may include the following steps A1 to A3:

[0102] A1. Based on the partial aggregation operation of the aggregation node, generate a partial aggregation node.

[0103] A new partial aggregation node is generated based on the partial aggregation operation in the aggregation node. The purpose of this partial aggregation node is to perform preliminary grouping and aggregation of the input data to reduce the data size of subsequent join operations.

[0104] A2. Insert the aggregated node between the connecting node and its child nodes.

[0105] The partial aggregation nodes generated in step A1 are inserted into the logical plan, specifically between the join node and its child nodes. By inserting partial aggregation nodes before the join, the join operation only processes the data that has already undergone partial aggregation, rather than the original complete dataset.

[0106] A3. Remove part of the aggregation operation of the aggregation node.

[0107] Some aggregation operations have been removed from the aggregation nodes because they have been "pushed down" to the partial aggregation nodes for execution. That is, the aggregation node can now focus solely on completing the final global aggregation operation, without needing to handle partial aggregation logic.

[0108] In the above implementation, the three steps (A1 to A3) effectively push some aggregation operations down from the aggregation node to before the connection node. By performing local aggregation on the data before connection, the amount of data involved in the connection can be significantly reduced, thereby reducing network transmission overhead. Connection operations are typically computationally intensive tasks, and reducing the amount of input data directly improves the efficiency of connection operations. Since partial aggregation is completed in an early stage, the final global aggregation operation becomes simpler and more efficient.

[0109] Optionally, in one specific embodiment, the pushdown operation on the original logic plan can be performed as follows: Figure 1c As shown. Figure 1c A schematic block diagram is shown, in which a partial aggregation operation of an aggregation node is inserted as a partial aggregation node between a connection node and its child nodes.

[0110] Figure 1c Figure (1) shows a schematic block diagram of the original logic plan. Figure 1c The original logical plan shown in (1) is as follows:

[0111]

[0112] In the above logical plan, the left child node Scan(table1) and the right child node Scan(table2) read data from table1 and table2 respectively. The Join node joins the two tables based on the condition t1.key = t2.key. The Project node selects the required columns (e.g., t1.key and t2.value2). The Aggregate node groups the results after the join.

[0113] Figure 1c (2) shows a schematic block diagram of the optimized logic plan. Figure 1c The logical plan shown in (2) is as follows:

[0114]

[0115] In the above logical plan, the Partial Aggregate node performs partial aggregation operations on the data in table1 and table2 respectively. The Join node joins the partial aggregation results of the two tables based on the condition t1.key = t2.key. The Projection node selects the required columns. The Aggregate node completes the final aggregation (global grouping) operation.

[0116] Assume the data in table1 and table2 are as follows:

[0117] Table 1 data:

[0118] Key Value1 k1 v1 k1 v2 k2 v3 k2 v4

[0119] Table 2 data:

[0120] Key Value2 k1 w1 k1 w2 k2 w3 k2 w4

[0121] Performing partial aggregation using GROUP BY key on table1 yields the following result:

[0122]

[0123]

[0124] Performing partial aggregation using GROUP BY key on table2 yields the following result:

[0125] Key k1 k2

[0126] Based on the condition t1.key = t2.key, the partial aggregation results of the two tables are joined ON t1.key = t2.key, and the result is:

[0127] Key k1 k2

[0128] Select the desired columns (e.g., t1.key and t2.value2) using the query `SELECT t1.key, t2.value2`. The result will be:

[0129] Key Value2 k1 w1 k2 w3

[0130] The projected results are then grouped using the final grouping operation `GROUP BY t1.key`, resulting in:

[0131] Key k1 k2

[0132] Based on the above examples, it can be seen that the optimized logical plan has the following advantages over the original logical plan:

[0133] 1) Reduced data volume

[0134] Original data volume: table1: 4 records; table2: 4 records.

[0135] Total: 8 records.

[0136] Data volume after partial aggregation: Table 1 partial aggregation result: 2 records; Table 2 partial aggregation result: 2 records.

[0137] Total: 4 records.

[0138] 50% of the data volume was reduced.

[0139] 2) Reduced network transmission costs

[0140] In distributed systems, join operations typically require redistributing data across different nodes (Shuffle operation). By performing partial aggregation operations on the two tables separately, the amount of data that needs to be transferred during the Shuffle process can be significantly reduced.

[0141] Optionally, in one implementation, the preset aggregation conditions include the type of the aggregation node. Accordingly, step 130 may include:

[0142] If the child node of the connecting node is not an aggregation node, and the aggregation node is of type responsible for grouping data and does not include aggregation functions, then the partial aggregation operation is pushed down from the aggregation node to before the connecting node.

[0143] An aggregation node is a type responsible for grouping data and does not include aggregation functions (such as SUM, COUNT, AVG, etc.). In other words, this aggregation node may simply deduplicate or group the input data according to a specified key. The above aggregation node type can be represented as grouponly. If an aggregation node is of type grouponly (i.e., only responsible for grouping and not including aggregation functions), it is safe to push some aggregation operations down to before the join nodes. This is because the role of a grouponly aggregation node is deduplication or grouping; performing some aggregation operations in advance can significantly reduce the amount of data involved in the join.

[0144] If the aggregation node is of type grouponly, pushing down partial aggregation operations before the join operation can significantly reduce the amount of data involved in the join, thereby reducing network transmission and computational overhead. For example, suppose a table has 1,000,000 rows of data, but the grouping key has only 10,000 unique values. By performing partial aggregation operations in advance, the amount of data involved in the join can be reduced from 1,000,000 rows to 10,000 rows.

[0145] If the aggregation node contains aggregation functions (such as SUM), pushing down Partial Aggregate may not significantly reduce the data volume. For example, even if partial aggregation operations are performed, the results of the aggregation functions still need to be retained. Therefore, the data volume may not be significantly reduced. Partial aggregation operations may also create redundancy with the final aggregation operation, leading to duplicate computations. For instance, the partial sum calculated in the pre-aggregation phase and the global sum in the final aggregation phase may involve performing the same addition operation repeatedly.

[0146] For example, let's suppose there is a structured query statement like the following:

[0147] SELECT t1.key, t2.value

[0148] FROM table1 t1

[0149] JOIN table2 t2

[0150] ON t1.key=t2.key

[0151] GROUP BY t1.key;

[0152] The logical plan for the above structured query statement can be as follows:

[0153]

[0154] In the example above, the aggregation node is of type Grouponly because it is only responsible for grouping and does not contain any aggregation functions. Therefore, some aggregation operations can be pushed down to before the connecting nodes.

[0155] In one specific implementation, some aggregation operations can be pushed down between the join node and one of the data tables. Taking table1 as an example, the logical plan after the pushdown is as follows:

[0156]

[0157] In the example above, by grouping and deduplicating the data in table1 beforehand, the amount of data involved in the join can be significantly reduced.

[0158] In another specific implementation, some aggregation operations can be pushed down between the join node and the two data tables, resulting in the following logical plan:

[0159]

[0160] In the example above, the Partial Aggregate node performs partial aggregation operations on the data in table1 and table2 respectively. The Join node joins the partial aggregation results of the two tables based on the condition t1.key = t2.key. Finally, the Aggregate node completes the global grouping operation.

[0161] For aggregate nodes whose type is not Grouponly, the following example illustrates this:

[0162] SELECT t1.key,SUM(t1.value),t2.value

[0163] FROM table1 t1

[0164] JOIN table2 t2

[0165] ON t1.key=t2.key

[0166] GROUP BY t1.key;

[0167] The logical plan for the above structured query statement can be as follows:

[0168]

[0169] In this logical plan, the Aggregate node is used to group and aggregate the results of joining table1 and table2 (GROUP BY t1.key, SUM(t1.value)). The Join node is used to join table1 and table2 based on the condition t1.key = t2.key. The Scan node reads data from table1 and table2 respectively.

[0170] Suppose we attempt to push some aggregation operations down to before the join nodes, resulting in the following logical plan:

[0171]

[0172] In this revised plan, the Partial Aggregate node performs a preliminary aggregation of the data in table1 before the Join operation (e.g., calculating PARTIAL_SUM(value)). However, in the final Aggregate node, these preliminary aggregation results still need to be further aggregated (i.e., FINAL_SUM(part_sum)). This double aggregation process can lead to duplicate calculations. For example, assuming multiple records in table1 have the same key, the value of each group will be summed once in the Partial Aggregate stage, and then these intermediate results will be summed again in the Aggregate stage. This is essentially equivalent to performing a summation operation twice on the same set of data, adding unnecessary computational overhead.

[0173] Besides the issue of redundant calculations, some aggregation operations may not significantly reduce the amount of data involved in the joins for aggregation nodes with aggregation functions. The reasons are as follows:

[0174] When using aggregation functions (such as SUM, COUNT, AVG, etc.), even with partial aggregation, the output still needs to retain the summary value for each group. For example, suppose table1 has 1 million records but only 10,000 unique keys. Even if a partial aggregation operation (such as PARTIAL_SUM(value)) is performed on table1, the output will still contain 10,000 group keys and their corresponding summary values. This means that the data volume is not significantly reduced because each group needs to retain its summary value.

[0175] For aggregation nodes of type Grouponly (grouping only, excluding aggregation functions), partial aggregation operations can significantly reduce data volume. This is because, in this case, the output is simply a set of unique grouping keys, without needing to retain any summary values. Therefore, the data volume is typically reduced dramatically (e.g., from 1 million records to 10,000 unique keys). However, when aggregation nodes contain aggregation functions, the output still needs to retain the summary values ​​for each group, so the effect of reducing data volume is limited.

[0176] Alternatively, in another implementation, the preset aggregation conditions include the type of the aggregation node and the broadcast operation status. Accordingly, step 130 may include steps 131 to 132 as follows:

[0177] 131. If the child node of the connection node is not an aggregation node, and the type of the aggregation node is responsible for grouping data and does not include aggregation functions, then the broadcast operation status is determined based on the data status of the data table connected by the connection node. The broadcast operation status includes suitable for broadcasting and unsuitable for broadcasting.

[0178] Broadcast operation is a common optimization technique used to accelerate join operations. However, the applicability of broadcast operation depends on the state of the data (such as data size and distribution characteristics).

[0179] If the child nodes of the join node are not themselves aggregate nodes, it means that the current join operation is being performed on unaggregated data. In this case, the join operation may involve a large amount of data, so it is worth assessing whether broadcast operation is appropriate.

[0180] Broadcasting is an optimization technique for replicating a small dataset to all computing nodes. Optionally, in one implementation, step 131, "determining the broadcast operation status based on the data status of the data tables connected to the connected nodes," includes the following steps B1 to B3:

[0181] B1. Obtain the amount of data in the data table connected to the connection node.

[0182] It may be necessary to obtain the data volume information of the data table involved in the connection operation as a basis for determining whether the broadcast operation is appropriate.

[0183] Data volume typically refers to the number of records or bytes in a data table. In big data processing systems, record count is a common metric. Methods for obtaining data volume can include, but are not limited to: directly obtaining the number of records in a data table through metadata or statistical information. For example, distributed computing frameworks such as Hive or Spark often maintain statistical information about tables (such as the number of rows, file size, etc.). Data volume can also be dynamically calculated through sampling.

[0184] Let's assume the logical plan is as follows:

[0185] Aggregate(GROUP BY t1.key)

[0186] └──Join(ON t1.key=t2.key)

[0187] ├──Scan(table1)

[0188] └──Scan(table2)

[0189] Assuming table1 contains 10 million records and table2 contains 1,000 records, then in step B1, the data volume of table1 and table2 can be obtained as 10 million and 1,000, respectively.

[0190] B2. Compare the data volume of the data table with the data volume threshold.

[0191] The data size threshold is a preset value used to determine whether the data table is small enough for broadcast operations. The data size threshold can be set based on factors such as system memory, network bandwidth limitations, or developer experience. Let's assume the data size threshold is 10,000. It should be understood that the specific value of the data size threshold should not be construed as a limitation of this application. In some cases, the threshold may be dynamically adjusted based on the current cluster load.

[0192] For each data table, its data volume can be compared to a data volume threshold: if the data volume is less than the threshold, the data table is considered suitable for broadcasting. If the data volume is greater than or equal to the threshold, the data table is considered unsuitable for broadcasting.

[0193] Continuing with the example above: Table1 has 10 million data entries, which is far greater than the data volume threshold (i.e., 10,000), therefore Table1 is not suitable for broadcasting. Table2 has 1,000 data entries, which is less than the data volume threshold (i.e., 10,000), therefore it is suitable for broadcasting.

[0194] B3. If there is a target data table with a data volume less than the data volume threshold, then the target data table is determined to be a data table suitable for broadcasting.

[0195] Continuing with the example above: table1 contains 10 million data entries, making it unsuitable for broadcasting. table2 contains 1,000 data entries, which is less than the data volume threshold (i.e., 10,000). Therefore, table2 is determined to be a suitable table for broadcasting. In this case, a broadcast operation can be performed on table2, thereby speeding up the connection operation.

[0196] In the above implementation, broadcasting operations can avoid the shuffle process for large-scale data, thereby reducing network transmission costs. Broadcasting replicates smaller datasets to all compute nodes, allowing join operations to be performed locally, thus improving performance. Shuffle refers to the process where data is transferred from one node to another, or moved from one partition to another, when an operation requires redistributing data (e.g., grouping or merging data by a key). The shuffle process involves sorting, grouping, and repartitioning data, all of which incur significant overhead. The above process reduces the number of shuffle operations, thereby reducing computational power consumption and improving efficiency.

[0197] Optionally, in one specific embodiment, step B3 may specifically include the following steps B31 to B32:

[0198] B31. If there is a target data table with a data volume less than the data volume threshold, then determine the data distribution status of the target data table.

[0199] In step B31, the data distribution of the target data table can be further analyzed to determine whether the target data table is suitable for broadcast operations. Even if the data volume of the target data table is less than the data volume threshold, uneven data distribution may still lead to performance problems in broadcast operations.

[0200] Data distribution refers to the distribution of data records in a data table across different partitions or nodes. Data distribution includes: uniform data distribution and skewed data distribution.

[0201] Among them, uniform data distribution means that data records are distributed relatively evenly across partitions; skewed data distribution means that data records are concentrated on certain partitions, resulting in less data or even no data in other partitions.

[0202] The data distribution status can be obtained in any of the following ways: the distribution characteristics of the data table can be obtained through metadata or statistics. For example, distributed computing frameworks such as Hive or Spark usually maintain partition-level statistics (such as the number of records per partition); the data distribution status can also be dynamically calculated through sampling or other estimation methods.

[0203] Continuing with the example above, let's look at a logical plan example:

[0204]

[0205]

[0206] Assuming that table2 has 1,000 records, which is less than the data volume threshold, we proceed to the next step of analysis: checking the data distribution of table2.

[0207] Specifically, uniform data distribution requires that the difference in the number of records in each partition be within a reasonable range (for example, the ratio of the number of records in the largest partition to the number of records in the smallest partition should not exceed a certain ratio threshold, let's say that the ratio threshold is 1.1, or it can be other values ​​such as 1.3, etc.).

[0208] If the data records in table2 are evenly distributed across all partitions, then it is considered to have a uniform data distribution; if the data records in table2 are concentrated in a few partitions, then it is considered to have a skewed data distribution.

[0209] Suppose table2 contains 1,000 records, evenly distributed across 10 partitions, with each partition containing approximately 100 records. In this case, the data distribution of table2 can be considered uniform. Specifically, the scenario of 1,000 records evenly distributed across 10 partitions can be described as follows:

[0210] Partition 1: 95 entries

[0211] Partition 2: 103 entries

[0212] Partition 3: 98 entries

[0213] Partition 4: 101 entries

[0214] Partition 5: 97 entries

[0215] Partition 6: 102 entries

[0216] Partition 7: 99 entries

[0217] Partition 8: 100 entries

[0218] Partition 9: 104 entries

[0219] Partition 10: 101 entries

[0220] The maximum number of records in a partition is 104, and the minimum number of records in a partition is 95.

[0221] The ratio is approximately 1.09 (close to 1, indicating that the data is evenly distributed).

[0222] If table2 has 1,000 records, with 900 concentrated in one partition and the remaining nine partitions containing only a small number of records, then table2's data distribution can be considered skewed. Specifically, the case where 1,000 records are unevenly distributed across 10 partitions can be described as follows:

[0223] Partition 1: 900 entries

[0224] Partition 2: 10 entries

[0225] Partition 3: 5 lines

[0226] Partition 4: 15 entries

[0227] Partition 5: 10 entries

[0228] Partition 6: 5 entries

[0229] Partition 7: 10 entries

[0230] Partition 8: 15 entries

[0231] Partition 9: 10 entries

[0232] Partition 10: 20 entries

[0233] The maximum number of records in a partition is 900, and the minimum number of records in a partition is 5.

[0234] The ratio is 180 (much greater than 1, indicating a skewed data distribution).

[0235] B32. If the data distribution is uniform, then the target data table is determined to be a suitable data table for broadcasting.

[0236] In the above implementation, through the steps (B31 to B32), if the data volume of the target data table is determined to be less than the data volume threshold, its suitability for broadcast operations can be further determined based on the data distribution status of the target data table. Specifically, the distribution characteristics of the target data table can be obtained through statistical information or dynamic sampling to determine whether it is uniformly distributed. If the data distribution status of the target data table is uniformly distributed, it is determined to be a suitable data table for broadcasting; otherwise, it is determined to be unsuitable for broadcast operations. This more refined judgment mechanism enables a reasonable choice between broadcast operations and other optimization strategies (such as the aggregation operations in the following section), thereby improving query performance. Choosing an appropriate optimization strategy can significantly reduce network transmission and computational overhead, while avoiding performance bottlenecks caused by uneven data distribution.

[0237] Optionally, in one embodiment, after step 131, the present application embodiment may further include the following steps C1 to C2:

[0238] C1. If the broadcast operation status is suitable for broadcasting, then obtain the suitable broadcasting data table.

[0239] Continuing with the example above:

[0240] For logical plans:

[0241]

[0242] It has been determined that table2 contains 1,000 records (less than the data size threshold), and these 1,000 records are evenly distributed across 10 partitions. Therefore, table2 can be considered a suitable data table for broadcasting.

[0243] In step C1, the data content of table2 can be obtained and prepared to be broadcast to all worker nodes participating in the computation.

[0244] C2. Send the data from the appropriate broadcast data table to the worker nodes participating in the computation in the distributed system.

[0245] Worker nodes are the units in a distributed system responsible for the actual computation tasks. Each worker node can receive broadcast data and perform join operations with data tables on the other side locally. Broadcast operations allow smaller data tables to be accessed directly on each worker node, thus avoiding the shuffle process for large-scale data. Join operations can be completed locally, reducing network transmission costs.

[0246] In distributed computing frameworks (such as Apache Spark or Apache Flink), broadcast operations are typically implemented using a shared memory mechanism. The data to be broadcast can be serialized, sent to all worker nodes, and stored in the memory of each worker node.

[0247] Continuing with the example above: the data in table2 can be serialized and sent to all worker nodes in the distributed system. Each worker node will receive the complete table2 data and load it into memory.

[0248] Each worker node can locally join the data in table2 with the partitioned data in table1. Since the data in table2 is already available locally, the join operation requires no additional network transmission, thus significantly improving performance.

[0249] In the above implementation, without broadcasting, join operations might require redistributing data from table1 and table2 across different nodes (i.e., a shuffle operation), increasing network transmission costs. Broadcasting avoids the shuffle process, significantly reducing network bandwidth usage. It also allows smaller tables to be accessed directly on each worker node, accelerating join operations. This broadcasting mechanism significantly reduces network transmission costs, improves join efficiency, and simplifies data distribution complexity. The above implementation is based on a Cost-Based Optimizer (CBO).

[0250] 132. If the broadcast operation status is unsuitable for broadcasting, the partial aggregation operation is pushed down from the aggregation node to before the connection node.

[0251] In step 132, if the broadcast operation is deemed unsuitable for broadcasting, the amount of data involved in the connection can be reduced by pushing down some aggregation operations, thereby reducing network transmission and computational overhead. When broadcasting is unsuitable, connection operations may involve a large amount of data, leading to high network transmission costs and computational overhead. By pushing down some aggregation operations before the connection nodes, data can be grouped and deduplicated in advance, thereby reducing the amount of data involved in the connection.

[0252] For example, continuing with the example above:

[0253]

[0254] Assuming both table1 and table2 have large amounts of data (e.g., 10 million records each), the broadcast operation status is "not suitable for broadcasting." Some aggregation operations can be pushed down before the join nodes, generating the following optimized logical plan:

[0255]

[0256] In this optimized plan: the Partial Aggregate node will group and deduplicate the data in table1 before the Join operation. Assuming table1 has 10 million records but only 10,000 unique keys, the amount of data involved in the join is reduced from 10 million to 10,000. The final Aggregate node is still responsible for performing the global aggregation operation.

[0257] In the above implementation, when broadcast operations are unsuitable, partial aggregation pushdown can avoid performance bottlenecks caused by broadcasts. By performing partial aggregation operations in advance, the amount of data involved in the connection can be significantly reduced, thereby lowering network transmission costs.

[0258] Optionally, in one specific implementation, steps 110 to 130 can be implemented using an optimization rule called the Partial Aggregate Pushdown Rule. This rule is designed to follow Spark's pluggable architecture, allowing users to enable or disable this optimization feature through simple configuration without modifying the underlying code. Alternatively, users can choose whether to enable this optimization rule based on their actual needs. If this feature is required, simply add the corresponding parameters to the Spark configuration file.

[0259] Alternatively, in one implementation, please refer to [link to relevant documentation]. Figure 1d , Figure 1d A schematic flowchart illustrating the SQL query statement is shown, wherein:

[0260] SQL Query: This is the SQL query statement submitted by the user. It is the starting point of the entire query processing flow, where the user specifies the operation to be performed through the SQL statement.

[0261] Dataset: Represents the data source for which a query operation needs to be performed, typically data stored in a distributed file system or other data storage system.

[0262] Unresolved Logical Plan: This refers to the preliminary logical plan, which contains information such as table names and column names that have not yet been resolved.

[0263] Catalog: This represents the location where metadata information is stored, including table structure, indexes, and other information, used to resolve unresolved logical plans.

[0264] Logical Plan: This refers to the parsed logical plan, where table names, column names, and other information have been parsed.

[0265] Optimized Logical Plan: This refers to a logically optimized plan that incorporates more efficient query execution strategies.

[0266] Physical Plans: Multiple possible physical execution plans, each with a different execution strategy.

[0267] Cost Model: Used to evaluate the cost of different physical plans and select the physical plan with the lowest cost as the final execution plan.

[0268] Selected Physical Plan: The optimal physical execution plan selected based on the cost model.

[0269] Resilient Distributed Datasets (RDDs): The basic data abstraction in Spark, representing an immutable, partitioned collection of records that can be stored in parallel across a cluster.

[0270] Analysis phase: In this phase, Spark performs syntax analysis and type checking on the SQL query, generating an unresolved logical plan.

[0271] Logical Optimization Phase: In this phase, the logical plan is optimized to generate an optimized logical plan. Inserting partial aggregate pushdown rules occurs during this phase.

[0272] Physical Planning Phase: In this phase, the optimized logical plan is converted into specific physical execution plans, and the optimal execution strategy is selected.

[0273] Code Generation Phase: The selected physical plan is converted into specific executable code.

[0274] Optionally, a partial aggregate pushdown rule can be inserted during the logic optimization phase to optimize SQL queries involving aggregate nodes and join nodes, where the aggregate node is a subordinate node of the join node. The specific steps are as follows:

[0275] For SQL queries involving aggregate nodes and join nodes, where the aggregate node is a subordinate node of the join node, logical checks can be performed to determine whether the partial aggregate pushdown rule applies to the SQL query.

[0276] Alternatively, the judgment can be based on the following three conditions:

[0277] The first condition is that neither of the two child nodes of the connecting node can currently be an aggregator node. If the child node of the connecting node is already an aggregator node, it means that some aggregation operations have already been completed in a previous stage, and repeatedly pushing down PartialAggregate will not bring additional performance improvements.

[0278] The second condition is that the aggregation node in the SQL query is of type grouponly, meaning it does not contain aggregate functions. If the aggregation node is not of type grouponly (such as an aggregation node containing SUM or AVG aggregation operations), it may be necessary to retain more intermediate results. Pushing down the aggregation operations is not very meaningful, provides limited performance improvement, and may even slow down performance.

[0279] The third condition is: based on CBO statistics, if the connected node can be converted to BroadcastJoin in the partial computation process, the partial aggregate pushdown rule can be skipped, for the same reason as above. Conversely, if the connected node cannot be converted to BroadcastJoin in the partial computation process, the partial aggregate pushdown rule can be applied.

[0280] The following section will illustrate the effect of the partial aggregate pushdownRule optimization rule by comparing the logical plans before and after optimization in the example:

[0281] Let's assume the original logical plan is as follows:

[0282] Aggregate[d#3],[d#3]

[0283] +-Project[d#3]

[0284] +-Join Inner,(d#3=e#4)

[0285] :-LocalRelation <empty>,[d#3]

[0286] +-LocalRelation <empty>[e#4]

[0287] Analysis of the logic plan before optimization shows that:

[0288] For the Aggregate node: The final Aggregate operation is to group and aggregate the d#3 field. This is a typical Grouponly Aggregate (containing only GROUP BY and not involving any aggregation functions such as SUM or COUNT).

[0289] For the Join node: Inner Join is used, with the join condition d#3 = e#4. The left and right child nodes are two empty local relations, indicating that the data source is empty.

[0290] The execution order is as follows: Data is first read from two LocalRelations. Then, an Inner Join operation is performed to generate intermediate results. Finally, an Aggregate operation is performed on the Join results to complete the grouping and aggregation.

[0291] In the original plan, the final aggregation operation was performed directly on the join result without prior partial aggregation in the left and right child nodes of the join. If the data volume is large, this will significantly increase the amount of data processed by the join operation, increasing network transmission costs and computational overhead.

[0292] The optimized logic plan is as follows:

[0293] Aggregate[d#3],[d#3]

[0294] +-Project[d#3]

[0295] +-Join Inner,(d#3=e#4)

[0296] :-AggregatePart[d#3],[d#3]

[0297] :+-LocalRelation <empty>,[d#3]

[0298] +-AggregatePart[e#4],[e#4]

[0299] +-LocalRelation <empty>[e#4]

[0300] Analysis of the optimized logic plan above shows that:

[0301] For the newly added Partial Aggregate node: In the optimized Plan, the AggregatePart operation is introduced in the left and right child nodes of Join. The left child node performs partial aggregation on field d#3 (AggregatePart[d#3],[d#3]). The right child node performs partial aggregation on field e#4 (AggregatePart[e#4],[e#4]).

[0302] For the final Aggregate node: After the Join operation is completed, a final Aggregate operation (Aggregate[d#3],[d#3]) is still retained. This is the Final Aggregate, used to merge the results from the Join.

[0303] The execution order is as follows: Data is first read from two LocalRelations. Before the Join operation, partial aggregation operations are performed on the left and right child nodes respectively to reduce the amount of data involved in the Join. Then, an Inner Join operation is performed to generate intermediate results. Finally, the Join results are subjected to a final Aggregate operation.

[0304] Compared to the logical plan before and after optimization, by performing partial aggregation operations in the left and right child nodes of the Join operation in advance, the amount of data involved in the Join can be significantly reduced, which can greatly reduce network transmission costs and disk I / O overhead. The smaller amount of data reduces the computational complexity of the Join operation, thereby improving the overall query performance, while also avoiding the repeated processing of a large amount of redundant data in the final Aggregate operation.

[0305] Optionally, in one specific embodiment, after step 130, the embodiments of this application may further include the following steps 140 to 160:

[0306] 140. Obtain the amount of data from the partial aggregation results connected by the connection node, wherein the partial aggregation results are the processing results obtained after the data tables connected by the connection node undergo partial aggregation operations.

[0307] In step 140, the partial aggregation result connected to the connection node is the processing result obtained after the partial aggregation node pushes down to perform partial aggregation operation on the original data table.

[0308] Partial aggregation operations are the process of partially summarizing the original data table. Partial aggregation operations can include grouping and preliminary aggregation calculations.

[0309] Continuing with the example above, let's examine the logical plan:

[0310]

[0311] It can retrieve the amount of data generated by the partial aggregation results from the partial aggregation node.

[0312] Let's assume the original data table table1 has 10 million records. After some aggregation operations, it has 10,000 records with unique grouping keys.

[0313] 150. Compare the data volume of the partial aggregation result with the data volume threshold.

[0314] For partial aggregation results, their data volume can be compared with a data volume threshold: if the data volume is less than the threshold, the data table is considered suitable for broadcasting. If the data volume is greater than or equal to the threshold, the data table is considered unsuitable for broadcasting.

[0315] Continuing with the example above, let's take a data volume threshold of 10,000 as an example. For some aggregation results, the data volume is 10,000 records, which is less than the data volume threshold. Therefore, it is determined that some aggregation results are suitable for broadcasting.

[0316] 160. If the amount of data in the partial aggregation result is less than the data amount threshold, then the data of the partial aggregation result with a data amount less than the data amount threshold is sent to the worker nodes participating in the computation in the distributed system.

[0317] Continuing with the example above: the data content of a partial aggregation result can be serialized and sent to all worker nodes in the distributed system. Each worker node will receive the complete partial aggregation result and load it into memory.

[0318] Each worker node locally joins a portion of the aggregation results with the partitioned data in table2. Since some of the aggregation results are already available locally, the join operation requires no additional network transmission, thus significantly improving performance.

[0319] In the above implementation, after performing partial aggregation processing on the original data table to obtain the partial aggregation processing result, a further judgment can be made: whether the data volume of the partial aggregation result is less than the data volume threshold. If so, the partial aggregation result can be broadcast to further reduce the amount of computation.

[0320] The data processing method provided in this application is applicable to Adaptive Query Execution (AQE) scenarios. AQE is a dynamic optimization technique widely used in modern distributed computing frameworks (such as Apache Spark). It optimizes the query plan and improves query performance by dynamically adjusting runtime information during query execution. In AQE scenarios, partial aggregation pushdown can easily generate Broadcast Exchanges, thus replacing traditional Shuffle Exchanges. This replacement can significantly reduce network I / O and disk I / O, resulting in a huge performance improvement. Compared to Shuffle Exchanges, Broadcast Exchanges only need to send smaller datasets to all nodes at once, without redistributing large-scale data. Shuffle Exchanges typically need to write data to disk (such as intermediate result files), while Broadcast Exchanges can directly load data into memory, avoiding disk read / write operations. Broadcast Exchanges allow smaller datasets to be directly accessed on each worker node, thereby avoiding the Shuffle process for large-scale data and significantly improving the efficiency of join operations.

[0321] Furthermore, the embodiments of this application are highly versatile and can be widely applied to various database architectures and computing engines, such as:

[0322] MPP architecture: In the Massively Parallel Processing (MPP) architecture, the embodiments of this application can effectively reduce the communication overhead between nodes and improve the efficiency of distributed computing.

[0323] Modern data warehouses: Whether it is a cloud-based distributed data warehouse (such as Snowflake, BigQuery) or a locally deployed data warehouse (such as Greenplum, ClickHouse), the optimization ideas of the embodiments in this application can be used as a reference.

[0324] Traditional databases: Even traditional relational databases on a single machine (such as MySQL and PostgreSQL) can reduce computation and I / O overhead and improve query performance through similar optimization strategies.

[0325] In the data processing method provided in this application embodiment, the logical plan of a structured query statement is obtained. Based on the tree structure of the logical plan, a target sub-plan is identified. The target sub-plan includes aggregation nodes and connection nodes, and the connection nodes are subordinate nodes of the aggregation nodes. If the child node of the connection node is not an aggregation node, and the target sub-plan satisfies the preset aggregation conditions, some aggregation operations are pushed down from the aggregation node to before the connection node, so that some aggregation operations are performed on the data table connected by the connection node before the connection operation is executed. In this application embodiment, the logical plan of a structured query statement can be obtained, and based on the tree structure of the logical plan, a target sub-plan including aggregation nodes and connection nodes, and the connection nodes are subordinate nodes of the aggregation nodes, can be identified. If the child node of the connection node of the target sub-plan is not an aggregation node, and the target sub-plan satisfies the preset aggregation conditions, some aggregation operations can be pushed down to before the connection node, thereby achieving: first, performing some aggregation operations on the data table to obtain some aggregation results; then performing connection operations on the partial aggregation results; and finally performing the final aggregation operation. Performing partial aggregation on the data packets before connection operations can significantly reduce the amount of data processed by the connection nodes, thereby avoiding the processing of duplicate and redundant information.

[0326] The embodiments of this application can reduce the computational burden.

[0327] In this embodiment, the method of this application embodiment will be described in detail. The data processing method provided in this application embodiment can be executed by an electronic device. In this embodiment, the electronic device is a server as an example for description, such as... Figure 2 As shown, the specific process of a data processing method is as follows:

[0328] 201. Obtain the logical plan of the structured query statement, wherein the logical plan reflects the logical operation steps of the structured query statement through a tree structure.

[0329] 202. Traverse the tree structure of the logical plan to obtain multiple aggregation nodes.

[0330] 203. For each of the aggregate nodes, obtain the node type of the lower-level nodes of the aggregate node.

[0331] 204. If there exists an aggregation node, and the node type of the subordinate node of the aggregation node is connection type, then the subordinate node of the aggregation node to the connection type is determined as the target sub-plan.

[0332] 205. If the child node of the connection node is not an aggregation node, and the type of the aggregation node is responsible for grouping data and does not include aggregation functions, then the broadcast operation status is determined based on the data status of the data table connected by the connection node. The broadcast operation status includes suitable for broadcasting and unsuitable for broadcasting.

[0333] Optionally, in one embodiment, after step 205, the present application embodiment may further include: if the broadcast operation status is suitable for broadcasting, then obtaining a suitable broadcasting data table; and sending the data of the suitable broadcasting data table to the worker nodes participating in the computation in the distributed system.

[0334] Optionally, in one implementation, "determining the broadcast operation status based on the data status of the data table connected by the connection node" may include:

[0335] Obtain the data volume of the data table connected by the connection node; compare the data volume of the data table with a data volume threshold; if there is a target data table with a data volume less than the data volume threshold, determine the data distribution status of the target data table; if the data distribution status is uniform, determine that the target data table is a suitable data table for broadcasting.

[0336] 206. If the broadcast operation status is unsuitable for broadcasting, generate partial aggregation nodes based on the partial aggregation operation of the aggregation nodes.

[0337] 207. Insert the aggregated node between the connecting node and its child nodes.

[0338] 208. Remove part of the aggregation operation of the aggregation node.

[0339] 209. Obtain the amount of data from the partial aggregation results connected by the connection node, wherein the partial aggregation results are the processing results obtained after the data tables connected by the connection node undergo partial aggregation operations.

[0340] 210. Compare the data volume of the partial aggregation result with the data volume threshold.

[0341] 211. If the amount of data in the partial aggregation result is less than the data amount threshold, then the data of the partial aggregation result with a data amount less than the data amount threshold is sent to the worker nodes participating in the computation in the distributed system.

[0342] The specific execution process of steps 201 to 211 has been explained in detail above, and will not be repeated here.

[0343] In the data processing method provided in this application embodiment, the logical plan of a structured query statement is obtained. Based on the tree structure of the logical plan, a target sub-plan is identified. The target sub-plan includes aggregation nodes and connection nodes, and the connection nodes are subordinate nodes of the aggregation nodes. If the child node of the connection node is not an aggregation node, and the target sub-plan satisfies the preset aggregation conditions, some aggregation operations are pushed down from the aggregation node to before the connection node, so that some aggregation operations are performed on the data table connected by the connection node before the connection operation is executed. In this application embodiment, the logical plan of a structured query statement can be obtained, and based on the tree structure of the logical plan, a target sub-plan including aggregation nodes and connection nodes, and the connection nodes are subordinate nodes of the aggregation nodes, can be identified. If the child node of the connection node of the target sub-plan is not an aggregation node, and the target sub-plan satisfies the preset aggregation conditions, some aggregation operations can be pushed down to before the connection node, thereby achieving: first, performing some aggregation operations on the data table to obtain some aggregation results; then performing connection operations on the partial aggregation results; and finally performing the final aggregation operation. Performing partial aggregation on the data packets before connection operations can significantly reduce the amount of data processed by the connection nodes, thereby avoiding the processing of duplicate and redundant information.

[0344] The embodiments of this application can reduce the computational burden.

[0345] The data processing method provided in this application pushes down partial aggregation operators (but not the final aggregation) based on join nodes and cost-based optimization (CBO) to achieve high-performance optimization for scenarios combining join and aggregation operations. This method delves into and explores how to more effectively process complex queries in distributed computing environments, especially queries containing join and aggregation operations, achieving significant performance improvements and resource utilization optimization.

[0346] The data processing method provided in this application has broad application prospects and significant market value. In the field of big data analytics, with the continuous growth of data volume and the increasing complexity of business needs, the demand for efficient query processing is also increasing. The data processing method provided in this application can significantly improve query efficiency, reduce response time, and provide users with faster data insight services. Furthermore, due to its flexibility and scalability, this technology can be further integrated into more distributed computing frameworks in the future, such as Apache Spark and Flink, to support more types of distributed computing tasks. This will greatly enhance the ability of these platforms to process large-scale data and may drive the development of next-generation data analysis tools and technologies.

[0347] To better implement the above methods, embodiments of this application also provide a data processing apparatus, such as... Figure 3 As shown, the device includes:

[0348] The logic plan acquisition unit 301 is used to acquire the logic plan of the structured query statement, wherein the logic plan reflects the logical operation steps of the structured query statement through a tree structure.

[0349] The sub-plan identification unit 302 is used to identify target sub-plans based on the tree structure of the logical plan, wherein the target sub-plan includes aggregation nodes and connection nodes, and the connection nodes are the subordinate nodes of the aggregation nodes, and the aggregation nodes are used to perform partial aggregation operations and final aggregation operations;

[0350] The operation pushdown unit 303 is used to push down the partial aggregation operation from the aggregation node to before the connection node if the child node of the connection node is not an aggregation node and the target sub-plan meets the preset aggregation conditions, so as to perform the partial aggregation operation on the data table connected by the connection node before the connection operation is executed.

[0351] In one implementation, the preset aggregation condition includes the type of the aggregation node; correspondingly, the operation pushdown unit 303 is specifically used to: if the child node of the connection node is not an aggregation node, and the type of the aggregation node is responsible for grouping data and does not include aggregation functions, then push down the partial aggregation operation from the aggregation node to before the connection node.

[0352] In one implementation, the preset aggregation condition further includes a broadcast operation state; correspondingly, the operation pushdown unit 303 is specifically used for:

[0353] If the child node of the connection node is not an aggregation node, and the type of the aggregation node is responsible for grouping data and does not include aggregation functions, then the broadcast operation status is determined based on the data status of the data table connected by the connection node. The broadcast operation status includes suitable for broadcasting and unsuitable for broadcasting.

[0354] If the broadcast operation status is unsuitable for broadcasting, the partial aggregation operation is pushed down from the aggregation node to before the connection node.

[0355] In one embodiment, operating the pushdown unit 303 includes:

[0356] A node generation subunit is used to generate partial aggregation nodes based on partial aggregation operations of the aggregation nodes;

[0357] A node insertion subunit is used to insert the partial aggregation node between the connecting node and the child node of the connecting node;

[0358] The operation removes the subunit, which is used to remove part of the aggregation operation of the aggregation node.

[0359] In one embodiment, the device further includes:

[0360] The data table acquisition unit is used to acquire the data table suitable for broadcasting when the broadcast operation status is suitable for broadcasting;

[0361] The data sending unit is used to send the data from the data table suitable for broadcast to the worker nodes participating in the computation in the distributed system.

[0362] In one embodiment, the operation of the push-down unit 303 further includes:

[0363] The table data volume acquisition subunit is used to acquire the data volume of the data table connected by the connection node.

[0364] The data volume comparison subunit is used to compare the data volume of the data table with the data volume threshold.

[0365] The suitable broadcast determination subunit is used to determine that the target data table is a suitable broadcast data table when there is a target data table with a data volume less than the data volume threshold.

[0366] In one implementation, broadcasting a designated subunit includes:

[0367] The distribution state sub-unit is used to determine the data distribution state of the target data table when there is a target data table with a data volume less than the data volume threshold.

[0368] The broadcast determination sub-unit is used to determine that the target data table is a suitable data table for broadcasting when the data distribution state is uniform.

[0369] In one embodiment, the sub-plan identification unit 302 includes:

[0370] The structure traversal subunit is used to traverse the tree structure of the logical plan to obtain multiple aggregate nodes;

[0371] The node type acquisition subunit is used to acquire the node type of the lower-level nodes of each aggregate node.

[0372] The sub-plan determination sub-unit is used to determine the sub-plan from the aggregation node to the lower-level node of the connection type when there is an aggregation node and the node type of the lower-level node of the aggregation node is connection type.

[0373] In one embodiment, the device further includes:

[0374] The data volume acquisition unit is used to acquire the data volume of the partial aggregation result connected by the connection node, wherein the partial aggregation result is the processing result obtained after the data table connected by the connection node undergoes a partial aggregation operation;

[0375] The comparison unit is used to compare the data volume of the partial aggregation result with the data volume threshold;

[0376] A broadcast unit is used to send data of the partial aggregation results whose data volume is less than the data volume threshold to the worker nodes participating in the computation in the distributed system if the data volume of the partial aggregation results is less than the data volume threshold.

[0377] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0378] In the data processing method provided in this application embodiment, the logical plan of a structured query statement is obtained. Based on the tree structure of the logical plan, a target sub-plan is identified. The target sub-plan includes aggregation nodes and connection nodes, and the connection nodes are subordinate nodes of the aggregation nodes. If the child node of the connection node is not an aggregation node, and the target sub-plan satisfies the preset aggregation conditions, some aggregation operations are pushed down from the aggregation node to before the connection node, so that some aggregation operations are performed on the data table connected by the connection node before the connection operation is executed. In this application embodiment, the logical plan of a structured query statement can be obtained, and based on the tree structure of the logical plan, a target sub-plan including aggregation nodes and connection nodes, and the connection nodes are subordinate nodes of the aggregation nodes, can be identified. If the child node of the connection node of the target sub-plan is not an aggregation node, and the target sub-plan satisfies the preset aggregation conditions, some aggregation operations can be pushed down to before the connection node, thereby achieving: first, performing some aggregation operations on the data table to obtain some aggregation results; then performing connection operations on the partial aggregation results; and finally performing the final aggregation operation. Performing partial aggregation on the data packets before connection operations can significantly reduce the amount of data processed by the connection nodes, thereby avoiding the processing of duplicate and redundant information.

[0379] The embodiments of this application can reduce the computational burden.

[0380] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0381] In some embodiments, the data processing apparatus may also be integrated into multiple electronic devices, such as multiple servers, with the data processing method of this application being implemented by the multiple servers.

[0382] In this embodiment, a server will be used as an example for detailed description. For example, ... Figure 4 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0383] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0384] The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402. In some embodiments, the processor 401 may include one or more processing cores; in some embodiments, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 401.

[0385] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing model training by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0386] The electronic device also includes a power supply 403 that supplies power to the various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0387] The electronic device may also include an input module 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0388] The electronic device may also include a communication module 405. In some embodiments, the communication module 405 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 405 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0389] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402, thereby realizing the various functions in the various method embodiments of this application, as follows:

[0390] Obtain the logical plan of the structured query statement, which reflects the logical operation steps of the structured query statement through a tree structure; based on the tree structure of the logical plan, identify the target sub-plan, wherein the target sub-plan includes aggregation nodes and connection nodes, and the connection node is a subordinate node of the aggregation node, the aggregation node is used to perform partial aggregation operations and final aggregation operations; if the child node of the connection node is not an aggregation node, and the target sub-plan meets the preset aggregation conditions, the partial aggregation operations are pushed down from the aggregation node to before the connection node, so that the partial aggregation operations are performed on the data tables connected by the connection node before the connection operations are executed.

[0391] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0392] Therefore, embodiments of this application provide a computer-readable storage medium storing instructions that can be loaded by a processor to perform steps in any of the data processing methods provided in embodiments of this application. For example, the instructions can perform the following steps:

[0393] Obtain the logical plan of the structured query statement, which reflects the logical operation steps of the structured query statement through a tree structure; based on the tree structure of the logical plan, identify the target sub-plan, wherein the target sub-plan includes aggregation nodes and connection nodes, and the connection node is a subordinate node of the aggregation node, the aggregation node is used to perform partial aggregation operations and final aggregation operations; if the child node of the connection node is not an aggregation node, and the target sub-plan meets the preset aggregation conditions, the partial aggregation operations are pushed down from the aggregation node to before the connection node, so that the partial aggregation operations are performed on the data tables connected by the connection node before the connection operations are executed.

[0394] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0395] According to one aspect of this application, a computer program product or computer program is provided, comprising instructions stored in a computer-readable storage medium. A processor of a computer device reads the instructions from the computer-readable storage medium and executes the instructions, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.

[0396] Since the instructions stored in the storage medium can execute the steps of any of the data processing methods provided in the embodiments of this application, the beneficial effects that any of the data processing methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0397] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0398] The foregoing has provided a detailed description of a data processing method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.< / empty> < / empty> < / empty> < / empty>

Claims

1. A data processing method, characterized in that, The method includes: Obtain the logical plan of the structured query statement, wherein the logical plan reflects the logical operation steps of the structured query statement through a tree structure; Based on the tree structure of the logical plan, a target sub-plan is identified, wherein the target sub-plan includes an aggregation node and a connection node, and the connection node is a subordinate node of the aggregation node. The aggregation node is used to perform partial aggregation operations and final aggregation operations. If the child node of the connection node is not an aggregation node, and the target sub-plan meets the preset aggregation conditions, the partial aggregation operation is pushed down from the aggregation node to before the connection node, so that the partial aggregation operation is performed on the data table connected by the connection node before the connection operation is executed.

2. The method as described in claim 1, characterized in that, The preset aggregation conditions include the type of the aggregation node; If the child node of the connecting node is not an aggregation node, and the target sub-plan satisfies a preset aggregation condition, the step of pushing down the partial aggregation operation from the aggregation node to before the connecting node includes: If the child node of the connecting node is not an aggregation node, and the aggregation node is of type responsible for grouping data and does not include aggregation functions, then the partial aggregation operation is pushed down from the aggregation node to before the connecting node.

3. The method as described in claim 1, characterized in that, The preset aggregation conditions include the type of aggregation node and the broadcast operation status; If the child node of the connecting node is not an aggregation node, and the target sub-plan satisfies a preset aggregation condition, the step of pushing down the partial aggregation operation from the aggregation node to before the connecting node includes: If the child node of the connection node is not an aggregation node, and the type of the aggregation node is responsible for grouping data and does not include aggregation functions, then the broadcast operation status is determined based on the data status of the data table connected by the connection node. The broadcast operation status includes suitable for broadcasting and unsuitable for broadcasting. If the broadcast operation status is unsuitable for broadcasting, the partial aggregation operation is pushed down from the aggregation node to before the connection node.

4. The method according to any one of claims 1-3, characterized in that, Before pushing the partial aggregation operation down from the aggregation node to the connection node, the following steps are included: Based on the partial aggregation operations of the aforementioned aggregation nodes, partial aggregation nodes are generated; Insert the aggregated node between the connecting node and its child nodes; Remove some aggregation operations from the aggregation node.

5. The method as described in claim 3, characterized in that, After determining the broadcast operation status based on the data status of the data table connected by the connection node, the method further includes: If the broadcast operation status is "suitable for broadcasting", then obtain the suitable broadcasting data table; The data from the appropriate broadcast data table is sent to the worker nodes participating in the computation in the distributed system.

6. The method as described in claim 3, characterized in that, Determining the broadcast operation status based on the data status of the data table connected by the connection node includes: Obtain the amount of data in the data table connected to the connection node; Compare the data volume of the data table with the data volume threshold; If there is a target data table with a data volume smaller than the data volume threshold, then the target data table is determined to be a data table suitable for broadcasting.

7. The method as described in claim 6, characterized in that, The step of determining that if there is a target data table with a data volume smaller than the data volume threshold, the target data table is suitable for broadcasting includes: If there is a target data table with a data volume smaller than the data volume threshold, then the data distribution status of the target data table is determined. If the data distribution is uniform, then the target data table is determined to be a suitable data table for broadcasting.

8. The method as described in claim 1, characterized in that, The tree structure based on the logical plan identifies the target sub-plan, including: The tree structure of the logical plan is traversed to obtain multiple aggregate nodes; For each of the aggregate nodes, obtain the node type of the lower-level nodes of the aggregate node; If there exists an aggregation node, and the node type of the subordinate node of the aggregation node is connection type, then the subordinate node of the aggregation node to the connection type is determined as the target sub-plan.

9. The method as described in claim 1, characterized in that, Before pushing down some aggregation operations from the aggregation node to the connection node if the child node of the connection node is not an aggregation node and the target sub-plan meets the preset aggregation conditions, in order to reduce the amount of data involved in the connection operations of the connection node, the method further includes: Obtain the amount of data from the partial aggregation results connected by the connection node, wherein the partial aggregation results are the processing results obtained after partial aggregation operations on the data tables connected by the connection node; The amount of data in the aggregated portion is compared with the data amount threshold. If the amount of data in the partial aggregation result is less than the data amount threshold, then the data in the partial aggregation result whose amount is less than the data amount threshold is sent to the worker nodes participating in the computation in the distributed system.

10. A data processing apparatus, characterized in that, The device includes: The logical plan acquisition unit is used to acquire the logical plan of the structured query statement. The logical plan reflects the logical operation steps of the structured query statement through a tree structure. The sub-plan identification unit is used to identify target sub-plans based on the tree structure of the logical plan, wherein the target sub-plan includes aggregation nodes and connection nodes, and the connection nodes are the subordinate nodes of the aggregation nodes. The aggregation nodes are used to perform partial aggregation operations and final aggregation operations. The operation pushdown unit is used to push down the partial aggregation operation from the aggregation node to before the connection node if the child node of the connection node is not an aggregation node and the target sub-plan meets the preset aggregation conditions, so as to perform the partial aggregation operation on the data table connected by the connection node before the connection operation is executed.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing instructions; the processor loads instructions from the memory to perform the steps of the data processing method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions adapted for loading by a processor to perform the steps of the data processing method according to any one of claims 1 to 9.

13. A computer program product, characterized in that, It includes instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 9.

Images