Data processing method and device, electronic equipment and storage medium

Abstract

The application discloses a data processing method and device, electronic equipment and storage medium, comprising: obtaining a structured query statement, the structured query statement comprising a plurality of aggregation statements; identifying an aggregation statement that does not need to be finally aggregated from the plurality of aggregation statements, the aggregation statement that does not need to be finally aggregated being a to-be-processed aggregation statement; obtaining an original aggregation logic plan corresponding to the to-be-processed aggregation statement; and replacing the original aggregation logic plan with a partial aggregation logic plan, so that when the partial aggregation logic plan is converted into a corresponding partial aggregation physical plan, a partial aggregation operation is performed, the partial aggregation logic plan corresponding to the partial aggregation physical plan, and the partial aggregation physical plan comprising a partial aggregation statement. According to the embodiment of the application, the aggregation statement that does not need to be finally aggregated is identified, and the corresponding logic plan is replaced, so that meaningless and power-wasting final aggregation operations are avoided, thereby saving computing resources.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In existing technologies, when performing data processing, aggregation operations typically involve two stages: partial aggregation and final aggregation. Partial aggregation first performs preliminary aggregation on local data at the data source or data partition to reduce the data volume, and then final aggregation calculates the final aggregation result across the entire dataset.

[0003] However, when dealing with certain queries or data distributions, the results of partial aggregation are often sufficient. Nevertheless, existing technologies still perform a final aggregation step on the results of partial aggregation, leading to a waste of computational resources. Summary of the Invention

[0004] This application provides a data processing method, apparatus, electronic device, and storage medium that can improve the problem of wasted computing resources in the prior art.

[0005] This application provides a data processing method, the method comprising: obtaining a structured query statement, wherein the structured query statement includes multiple aggregation statements; identifying aggregation statements from the multiple aggregation statements that do not require final aggregation, wherein the aggregation statements that do not require final aggregation are aggregate statements to be processed; obtaining the original aggregation logic plan corresponding to the aggregate statements to be processed, wherein the original aggregation logic plan corresponds to an original aggregation management plan, the original aggregation management plan includes partial aggregation statements and a final aggregation statement, the partial aggregation statements are used to perform partial aggregation operations, and the final aggregation statements are used to perform final aggregation operations; replacing the original aggregation logic plan with the partial aggregation logic plan, so that when the partial aggregation logic plan is converted into a corresponding partial aggregation management plan, the partial aggregation operation is executed, wherein the partial aggregation logic plan corresponds to the partial aggregation management plan, and the partial aggregation management plan includes the partial aggregation statements.

[0006] This application provides a data processing apparatus, the apparatus comprising:

[0007] The statement acquisition unit is used to acquire structured query statements, wherein the structured query statements include multiple aggregate statements;

[0008] The statement recognition unit is used to identify, from the plurality of aggregation statements, the aggregation statements that do not need to be finally aggregated, wherein the aggregation statements that do not need to be finally aggregated are the aggregation statements to be processed;

[0009] The original logic plan unit is used to obtain the original aggregation logic plan corresponding to the aggregation statement to be processed. The original aggregation logic plan corresponds to an original aggregation processing plan, which includes partial aggregation statements and final aggregation statements. The partial aggregation statements are used to perform partial aggregation operations, and the final aggregation statements are used to perform final aggregation operations.

[0010] A logic plan replacement unit is used to replace the original aggregation logic plan with a partial aggregation logic plan, so that when the partial aggregation logic plan is converted into a corresponding partial aggregation logic plan, the partial aggregation operation is executed, wherein the partial aggregation logic plan corresponds to the partial aggregation logic plan, and the partial aggregation logic plan includes the partial aggregation statement.

[0011] In one implementation, the statement recognition unit includes:

[0012] The logic plan subunit is used to obtain the original aggregation logic plan corresponding to each aggregation statement;

[0013] The aggregation state subunit is used to determine the aggregation state of the corresponding aggregation statement based on the original aggregation logic plan, wherein the aggregation state includes whether final aggregation is required or not.

[0014] In one implementation, the original aggregation logic plan includes aggregation nodes; correspondingly, the aggregation state subunit includes:

[0015] The grouping key state sub-unit is used to determine the grouping key state of the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs, based on the grouping expression of the aggregation node.

[0016] The aggregation state sub-unit is used to determine the aggregation state of the aggregation statement based on the grouping key state of the aggregation statement.

[0017] In one implementation, the group key state includes the absence of a group key;

[0018] Correspondingly, the group key state sub-unit is specifically used to: determine that when the group expression of the aggregation node is empty, the aggregation statement corresponding to the original aggregation logic plan to which the group expression belongs does not have a group key;

[0019] The aggregation state sub-unit is specifically used to: determine that the aggregation state of the aggregation statement needs to be finally aggregated when the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs does not have a grouping key.

[0020] In one implementation, the grouping key state includes the presence of a grouping key;

[0021] Correspondingly, the group key state sub-unit is specifically used to: determine that the aggregation statement corresponding to the original aggregation logic plan to which the group expression belongs has a group key when the group expression of the aggregation node is not empty;

[0022] The aggregation state sub-unit is specifically used to: determine that the aggregation state of the aggregation statement does not require final aggregation when the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs has a grouping key.

[0023] In one implementation, the aggregation state subunit includes:

[0024] The pre-partitioning sub-unit is used to check the pre-partitioning status of the data source in the original aggregation logic plan, wherein the data source is the data to be queried by the structured query statement;

[0025] The aggregation state sub-unit is used to determine the aggregation state of the aggregation statement based on the pre-partitioning state of the data source.

[0026] In one implementation, the pre-partitioning state includes pre-partitioning performed and pre-partitioning not performed;

[0027] The pre-partitioning sub-unit is specifically used to: determine that the pre-partitioning status has been pre-partitioned when the data in the data source is partitioned and stored according to the grouping key;

[0028] When the data in the data source is not partitioned and stored according to the grouping key, the pre-partitioning state is determined to be no pre-partitioning.

[0029] In one implementation, the aggregated state sub-unit is specifically used for:

[0030] If the pre-partitioning state is one in which pre-partitioning has already been performed, then the aggregation state is determined to be one in which final aggregation is not required.

[0031] If the pre-partitioning state is that no pre-partitioning has been performed, the aggregation state is determined to be that final aggregation is required.

[0032] In one implementation, the statement recognition unit includes:

[0033] The first determination subunit is used to determine that the aggregation state of the aggregation statement needs to be finalized when the aggregation statement does not have a grouping key.

[0034] The second determination subunit is used to determine that the aggregation state of the aggregation statement does not require final aggregation when the aggregation statement has a grouping key.

[0035] In one implementation, the statement recognition unit includes:

[0036] The subsequent execution subunit is used to obtain the subsequent execution statement corresponding to each of the aggregate statements;

[0037] A semantic understanding subunit is used to perform semantic understanding on the subsequent execution statements to obtain the semantic understanding result of the subsequent execution statements;

[0038] The aggregation state subunit is used to determine the aggregation state of the aggregation statement based on the semantic understanding result.

[0039] This application also provides an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute steps in any of the data processing methods provided in this application.

[0040] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the data processing methods provided in this application.

[0041] This application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps in any of the data processing methods provided in this application.

[0042] In the data processing method provided in this application embodiment, a structured query statement is obtained, which includes multiple aggregation statements. From these multiple aggregation statements, aggregation statements that do not require final aggregation (denoted as aggregation statements to be processed) are identified. The original aggregation logic plan of the aggregation statements to be processed is obtained, wherein the original aggregation logic plan corresponds to an original aggregation physical plan, and the original aggregation physical plan is used to perform partial aggregation operations and final aggregation operations. The original aggregation logic plan is replaced with a partial aggregation logic plan so that when the partial aggregation logic plan is converted into a corresponding partial aggregation physical plan, partial aggregation operations are performed, instead of the partial aggregation operations and final aggregation operations corresponding to the original aggregation physical plan.

[0043] In this embodiment, aggregate statements that do not require final aggregation are first identified from multiple aggregate statements. Then, the original aggregation logic plan corresponding to the aggregate statement to be processed is replaced with a partial aggregation logic plan, so that the aggregate statement to be processed performs partial aggregation operations, instead of performing both partial and final aggregation operations. By identifying aggregate statements that do not require final aggregation and replacing their corresponding logic plans, this embodiment avoids the execution of meaningless and computationally wasteful final aggregation operations, thereby saving computing resources. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1a This is a schematic diagram illustrating an application scenario of the data processing method provided in this application;

[0046] Figure 1b This is a flowchart illustrating the data processing method provided in an embodiment of this application;

[0047] Figure 1c A schematic block diagram showing the correspondence between the original aggregation logic plan and the original aggregation logic plan;

[0048] Figure 1d A schematic block diagram showing the correspondence between partial aggregation logic plans and partial aggregation logic plans is provided.

[0049] Figure 1e A schematic flowchart illustrating the SQL query statement is shown.

[0050] Figure 2 This is a flowchart illustrating a data processing method provided in another specific embodiment of this application;

[0051] Figure 3 This is a schematic diagram of a data processing apparatus provided in one embodiment of this application;

[0052] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] This application provides a data processing method, apparatus, electronic device, and storage medium.

[0055] Specifically, the data processing device can be integrated into an electronic device, such as a terminal or server. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal and server can be directly or indirectly connected via wired or wireless communication; this application does not impose any restrictions on this connection.

[0056] In some embodiments, the data processing apparatus may also be integrated into multiple electronic devices, such as multiple servers, with the data processing method of this application being implemented by the multiple servers.

[0057] In some embodiments, the terminal can also be used as a server to implement some or all of the functions of a server.

[0058] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0059] Please see details Figure 1aThe method provided in this application embodiment may include: obtaining a structured query statement, wherein the structured query statement includes multiple aggregation statements; identifying aggregation statements that do not require final aggregation from the multiple aggregation statements, wherein the aggregation statements that do not require final aggregation are aggregation statements to be processed; obtaining the original aggregation logic plan corresponding to the aggregation statements to be processed, wherein the original aggregation logic plan corresponds to an original aggregation management plan, the original aggregation management plan includes partial aggregation statements and a final aggregation statement, the partial aggregation statements are used to perform partial aggregation operations, and the final aggregation statements are used to perform final aggregation operations; replacing the original aggregation logic plan with the partial aggregation logic plan, so that when the partial aggregation logic plan is converted into a corresponding partial aggregation management plan, the partial aggregation operation is executed, wherein the partial aggregation logic plan corresponds to the partial aggregation management plan, and the partial aggregation management plan includes the partial aggregation statements.

[0060] In the above method, aggregate statements that do not require final aggregation are first identified from multiple aggregate statements. Then, the original aggregation logic plan corresponding to the aggregate statement to be processed is replaced with a partial aggregation logic plan, so that the aggregate statement to be processed performs partial aggregation operations, rather than performing both partial and final aggregation operations. This embodiment of the application, by identifying aggregate statements that do not require final aggregation and replacing their corresponding logic plans, avoids the execution of meaningless and computationally wasteful final aggregation operations, thereby saving computational resources.

[0061] The primary application scenario of this application is optimizing the logical plan of SQL queries when Spark performs multi-join computations. In the field of big data processing, Apache Spark is a widely used distributed computing engine, whose core advantage lies in its ability to efficiently process large-scale datasets. However, when dealing with complex SQL queries (such as multi-table joins and aggregation operations), Spark's default execution plan may suffer performance degradation due to redundant computation steps and unnecessary data transfer. Therefore, this application proposes an optimization method for the logical plan of SQL queries, aiming to improve the overall execution efficiency of the job by reducing the computational overhead between partial aggregation and final aggregation.

[0062] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent should be explicitly obtained, and the collection, use and processing of related data should comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0063] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0064] In this embodiment, a data processing method is provided. For example... Figure 1b As shown, the specific process of this method may include the following steps 110 to 140:

[0065] 110. Obtain a structured query statement, wherein the structured query statement includes multiple aggregate statements.

[0066] Structured Query Language (SQL) is a programming language specifically designed for managing and manipulating relational databases. As one of the most important and widely used languages ​​in the database field, SQL provides a standardized set of methods that enable users to interact with databases efficiently. Through SQL, users can perform a variety of complex operations, including but not limited to querying data, inserting new records, updating existing records, deleting records, and creating or modifying the database structure. These capabilities make SQL an indispensable tool in data management, data analysis, and system development.

[0067] SQL, as the standard language for interacting with databases, is widely used in various Database Management Systems (DBMS), such as MySQL, PostgreSQL, Oracle Database, Microsoft SQL Server, and SQLite. While these DBMSs may differ in their underlying implementations, they all support SQL as their core query language, ensuring high compatibility and portability. Furthermore, with the development of big data and cloud computing technologies, the application scope of SQL has further expanded. Many modern data processing frameworks (such as Apache Hive and Spark SQL) also support SQL syntax, enabling users to leverage the familiar SQL language to process large-scale datasets stored in distributed environments.

[0068] Aggregate statements are SQL statements used to perform aggregation operations. Aggregation is an operation that summarizes and calculates a set of data. Its purpose is to process large amounts of data through specific mathematical or logical operations, ultimately generating a single result value. Aggregate operations have wide applications in data analysis and statistics, such as calculating sums, averages, maximums, minimums, and counts, and are often used to support business decisions, report generation, and data mining scenarios.

[0069] Aggregation operations are typically performed in two phases: partial aggregation and final aggregation. This phased design aims to optimize performance, reduce data transfer volume, and improve the overall efficiency of the system.

[0070] Partial aggregation refers to a local aggregation step that occurs on different nodes or partitions of data storage during distributed query execution. Since data is typically distributed across multiple nodes or partitions, each node first performs preliminary aggregation calculations on its locally stored data to reduce the amount of data that needs to be transmitted over the network. For example, in a distributed system, if a query requires calculating the sum of all data, each node will first calculate the partial sum of its local data and then pass these partial sums as intermediate results to the next step. In this way, partial aggregation can significantly reduce network bandwidth consumption and speed up subsequent computations.

[0071] Final aggregation refers to the process of globally summarizing the local aggregation results generated by each node or partition after all partial aggregations have been completed. The task of this stage is to further merge all intermediate results to obtain the global aggregation result for the entire dataset. For example, in the above example of calculating the sum, the final aggregation stage will add the partial sums of each node to obtain the sum of the entire dataset. Final aggregation is usually performed by a central node or coordinator to ensure that all local results are correctly included in the calculation.

[0072] It's important to note that not all queries require final aggregation. In certain scenarios, the results of partial aggregation may be sufficient, eliminating the need for further final aggregation. For example, when a query focuses only on local statistics for each partition, the results of partial aggregation can be used directly without additional summarization.

[0073] 120. From the plurality of aggregation statements, identify the aggregation statements that do not require final aggregation, wherein the aggregation statements that do not require final aggregation are the aggregation statements to be processed.

[0074] Optionally, multiple aggregation statements can be identified and processed to filter out specific statements that do not require final aggregation. The specific identification process will be described in detail below.

[0075] Optionally, in one embodiment, step 120 may specifically include the following steps A1 to A2:

[0076] A1. For each of the aggregation statements, obtain the original aggregation logic plan corresponding to the aggregation statement.

[0077] The original aggregation logic plan is the original logic plan corresponding to the aggregation statement. This logic plan describes the complete flow and logical structure of the aggregation operation.

[0078] A logical plan is an abstract representation of an SQL query, providing a high-dimensional perspective on the query's semantics and operational flow. This representation focuses on expressing the user's intent and the logical steps of data processing, without delving into the specific details of how these steps are actually executed in the underlying database system. A logical plan is a mapping or transformation of the SQL query's semantics. This means that a logical plan accurately reflects the user's query intent, such as which tables to retrieve data from, which filtering conditions to apply, and which aggregation operations to perform. In this way, the logical plan ensures a true representation of the user's query intent.

[0079] The logical plan focuses on defining the logical structure of the query, including but not limited to the selection of data sources (e.g., tables and views), the data join methods (e.g., inner join, left join), the application of filtering conditions (WHERE clause), grouping and sorting rules (GROUP BY and ORDER BY), and aggregation operations (e.g., SUM, COUNT). While the logical plan details these operations, it does not specify which algorithm or technique should be used to perform them. For example, the logical plan might indicate that a table join operation is needed, but it will not explicitly specify whether the table join operation should use a nested loop join or a hash join.

[0080] A2. Based on the original aggregation logic plan, determine the aggregation state of the corresponding aggregation statement, wherein the aggregation state includes those that require final aggregation and those that do not require final aggregation.

[0081] In this embodiment of the application, it can be determined whether the aggregation statement corresponding to the original aggregation logic plan needs to be finally aggregated or not, based on the original aggregation logic plan. Step A2 is described in detail below.

[0082] In the above implementation, determining the aggregation state of an aggregation statement based on a logical plan is an efficient and intuitive method. This method can deeply analyze the structural and semantic information of the query, thus providing a clear basis for optimization decisions. Specifically, by parsing the logical plan, the key structural features of the query can be clearly detected, including but not limited to the following aspects:

[0083] 1. Does it contain a GROUP BY clause?

[0084] The GROUP BY clause is a crucial syntactic element in SQL queries used for grouping data. Its presence or absence directly impacts how aggregation operations are executed. If a query contains a GROUP BY clause, it indicates that the data needs to be grouped and statistically analyzed according to a specific grouping key; conversely, the absence of a GROUP BY clause usually signifies a global aggregation operation on the entire dataset. Analyzing the logical plan can quickly identify the presence of a GROUP BY clause, thus determining the specific form of the aggregation operation.

[0085] 2. Is the query targeting local or global results?

[0086] The target of a query determines the scope and depth of the aggregation operation. For some queries, users may only be interested in local statistics (such as the results of a specific partition or node) and do not need a global summary. In this case, the results of partial aggregation are sufficient to meet the needs, and there is no need to perform the final aggregation. Other queries, however, explicitly require global results and need to merge the partial aggregation results of all partitions to generate the final answer. Through logical plans, it is possible to clearly distinguish whether the query target is local or global, thereby more accurately determining the aggregation state of the aggregation statement.

[0087] 3. What is the grouping key for the data?

[0088] A grouping key is a field or expression specified in the GROUP BY clause that defines how data is divided into different groups. The choice of grouping key not only affects the granularity of aggregation operations but also determines how data is distributed and computed. For example, in a distributed system, the grouping key may be associated with a data partitioning strategy, thus affecting the execution efficiency of partial aggregations. By parsing the logical plan, information about the grouping key can be extracted, and combined with the query context, it can be determined whether the requirements can be directly met through partial aggregation.

[0089] Based on the structural information of the query above, the aggregation status of aggregation statements can be determined more clearly and intuitively. Specifically, by analyzing the logical plan, it is possible to accurately determine which aggregation statements need to perform the final aggregation and which can complete the task by relying only on partial aggregation. This determination not only ensures the correctness of the query results but also significantly reduces unnecessary computational overhead and improves overall performance.

[0090] Optionally, in one implementation, the original aggregation logic plan includes aggregation nodes. Accordingly, step A2 may specifically include steps B1 to B2 as follows:

[0091] B1. Based on the grouping expression of the aggregation node, determine the grouping key state of the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs.

[0092] A key step in optimizing SQL queries is analyzing the grouping expressions in each aggregation node. Grouping expressions define how data is divided into different groups and determine which field or expression values ​​will be used to calculate local or global aggregation results. By examining these grouping expressions, the grouping key status of the corresponding aggregation statement in the original aggregation logic plan can be determined. Grouping key status includes: no grouping key and a grouping key exists.

[0093] Empty grouping expression: When the grouping expression of an aggregation node is empty, it means that the current aggregation operation does not depend on any specific grouping key; that is, the grouping key status is: no grouping key exists. This situation usually occurs in global aggregation operations, such as calculating the sum or average of the entire dataset.

[0094] Grouping expression not empty: If an aggregation node contains a non-empty grouping expression, it means the data will be grouped according to the specified field or expression, i.e., the grouping key status is: a grouping key exists. In this case, the aggregation operation will calculate the aggregation result separately for each group, such as the total number of employees in each department or the total sales of each product.

[0095] B2. Determine the aggregation state of the aggregation statement based on the grouping key state of the aggregation statement.

[0096] Continuing with the example above, if the grouping key status is "no grouping key," it means that final aggregation is required. If no grouping key exists, the goal of the current aggregation operation is to perform a global aggregation on the entire dataset. In this case, a final aggregation step is usually required to ensure that all data is correctly summarized. Therefore, for this type of aggregation statement, its aggregation status should be determined as "final aggregation required."

[0097] A grouping key status of "grouping key exists" indicates that final aggregation is unnecessary. If the aggregation node contains a valid grouping expression, it means the data has been reasonably divided into multiple groups, and the aggregation results of each group are sufficient to meet the query requirements. In this case, the results of partial aggregation can be used directly without performing a final aggregation step. Therefore, in this situation, the aggregation status of the aggregation statement should be determined as "final aggregation not required".

[0098] Optionally, in one specific implementation, the group key state includes the absence of a group key.

[0099] Accordingly, step B1 may include: if the grouping expression of the aggregation node is empty, then determine that the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs does not have a grouping key.

[0100] When parsing the logical plan, if a grouping expression for an aggregation node is found to be empty, it means that the current aggregation operation does not specify a grouping condition. In other words, the goal of this aggregation operation is to globally summarize all input data without needing to partition the data based on specific fields or expressions. Therefore, it can be determined that this aggregation statement does not contain any grouping keys, and its purpose is to generate a single result value that covers all the data.

[0101] Step B2 may include: if the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs does not have a grouping key, determining that the aggregation state of the aggregation statement is that final aggregation is required.

[0102] Since the aggregation statement does not specify a grouping key, it means that the entire dataset needs to be aggregated to produce the final result. Therefore, in this case, the final aggregation step can be retained to ensure that the results of all local aggregations are correctly merged to obtain the global aggregation result.

[0103] Alternatively, in another specific implementation, the group key state includes the presence of a group key.

[0104] Accordingly, step B1 may include: if the grouping expression of the aggregation node is not empty, then determine that the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs has a grouping key.

[0105] When examining aggregation nodes in a logical plan, if their grouping expressions are not empty, it indicates that the aggregation operation groups data based on certain specific fields or expressions. Further analysis of the grouping expressions can identify the specific grouping keys and confirm that the aggregation statement does indeed involve data grouping.

[0106] Step B2 may include: if the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs has a grouping key, determining that the aggregation state of the aggregation statement is that no final aggregation is required.

[0107] Since the aggregation statement has already specified the grouping key, and the results of the local aggregation can satisfy the query requirements, there is no need to perform an additional final aggregation step. This not only reduces computational overhead but also improves query efficiency. For example, in a distributed database environment, each node can independently complete its local aggregation task without passing intermediate results to the central node for global aggregation.

[0108] Alternatively, in another embodiment, step A2 may specifically include the following steps C1 to C2:

[0109] C1. In the original aggregation logic plan, check the pre-partitioning status of the data source, wherein the data source is the data to be queried by the structured query statement.

[0110] In optimizing SQL query aggregation operations, besides optimizing based on the analysis of aggregation nodes and their grouping expressions, the aggregation process can also be optimized based on the pre-partitioning status of the data source. The pre-partitioning status of the data directly affects whether the aggregation process can be simplified, thereby avoiding unnecessary final aggregation.

[0111] Optionally, in one specific embodiment, the pre-partitioning state includes pre-partitioning performed and no pre-partitioning performed. Accordingly, step C1 may specifically include the following steps C11 to C12:

[0112] C11. If the data in the data source is partitioned and stored according to the grouping key, then the pre-partitioning state is determined to be pre-partitioned.

[0113] Optionally, it can be verified whether the data in the data source has been partitioned and stored according to the grouping key. If the data has been reasonably divided into multiple subsets, and the data in each subset shares the same grouping key value, then the data source is in a "pre-partitioned" state.

[0114] In this scenario, since the data is already physically organized according to the grouping key, the results of local aggregation can be used directly without further merging of the global results. For example, in a distributed database system, each node may only contain data with a specific grouping key, allowing local aggregation tasks to be completed independently without the need to transfer data across nodes for the final aggregation.

[0115] C12. If the data in the data source is not partitioned and stored according to the grouping key, then the pre-partitioning status is determined to be that no pre-partitioning has been performed.

[0116] If the data in the data source is not partitioned according to the grouping key, that is, the data is randomly distributed or organized according to other non-grouping key standards, then the pre-partitioning status of the data source should be judged as "not pre-partitioned".

[0117] In this scenario, although local aggregations can be performed separately on each node, since the data is not organized according to the grouping key, a final aggregation step is needed to summarize all local results to ensure the correct global aggregation result is obtained.

[0118] C2. Based on the pre-partitioning state of the data source, determine the aggregation state of the aggregation statement.

[0119] After clarifying the pre-partitioning state of the data source, the next step is to determine the aggregation state of the aggregation statement based on this information. This step helps identify which aggregation operations can directly meet the requirements through local aggregation, thereby avoiding unnecessary final aggregation steps.

[0120] Optionally, in one specific embodiment, step C2 may specifically include the following steps C21 to C22:

[0121] C21. If the pre-partitioning state is one in which pre-partitioning has already been performed, determine that the aggregation state is one in which final aggregation is not required.

[0122] When the data source is already partitioned and stored according to the grouping key, this means that the results of local aggregations can accurately reflect the statistics of each group. In this case, the system can directly use the results of these local aggregations without performing additional final aggregation steps.

[0123] For example, when calculating the total number of employees in each department, if the data has been partitioned and stored according to department ID, then each node only needs to count its local data, without having to aggregate the results to the central node for global counting.

[0124] C22. If the pre-partitioning state is that no pre-partitioning has been performed, determine that the aggregation state is that final aggregation is required.

[0125] If the data source is not partitioned and stored according to the grouping key, the results of local aggregations can only provide a partial view and cannot directly generate a complete global result. To obtain accurate global statistics, a final aggregation step needs to be performed to merge all local results together.

[0126] For example, suppose you want to calculate the average salary of all employees in a company, but the data is not stored in partitions according to department or other relevant fields. In this case, each node first calculates a partial average of its local data, and these partial results need to be further aggregated to arrive at the average salary for the entire company.

[0127] In the above implementation, by deeply analyzing the pre-partitioning state of the data source, the execution strategy of aggregation operations can be optimized more accurately. This method not only improves system performance but also reduces unnecessary computational overhead and increases query efficiency. Specifically: when the data in the data source is already partitioned and stored according to the grouping key, the results of local aggregation can be used directly, usually without the need for a final aggregation step. If the data in the data source is not partitioned and stored according to the grouping key, then a final aggregation step is required to summarize all local results to ensure the correct global aggregation result. This optimization strategy based on the pre-partitioning state of the data source enables the database management system to significantly reduce resource consumption and improve response speed while ensuring query accuracy.

[0128] Alternatively, in another embodiment, step 120 may specifically include the following steps D1 to D2:

[0129] D1. If the aggregation statement does not have a grouping key, the aggregation state of the aggregation statement is determined to be that final aggregation is required.

[0130] In the process of optimizing SQL queries, the grouping key of an aggregation statement is one of the important factors determining the aggregation state of that statement. By analyzing whether an aggregation statement has a grouping key, it is possible to effectively determine whether the final aggregation needs to be performed.

[0131] The absence of a grouping key in an aggregation statement typically occurs in operations that perform global statistics on the entire dataset. For example, when calculating sums (SUM), averages (AVG), maximums (MAX), minimums (MIN), or counts (COUNT), if no GROUP BY clause is specified, it means that all data needs to be summarized to generate a single global result.

[0132] Step D1 above can be achieved by parsing the logical plan. Specifically, when parsing the logical plan, if the grouping expression of the aggregation node is found to be empty (i.e., no grouping key is defined), it can be clearly understood that the goal of the aggregation operation here is to perform a global aggregation on the entire dataset. In this case, partial aggregation alone cannot meet the requirements, because the result of partial aggregation is usually a statistical value of local data, while the global result needs to further merge all local results. Therefore, the aggregation status of such aggregation statements can be determined as "final aggregation is required". For example, in the scenario of calculating the total salary of all employees in a company, since there is no grouping by department or other fields, the data of all employees needs to be summarized together. Another example is in the scenario of calculating the total number of records in a table; regardless of which partition or node the data is distributed on, all local count results need to be added together to obtain the global total number of records.

[0133] D2. If the aggregation statement has a grouping key, determine that the aggregation state of the aggregation statement is that no final aggregation is required.

[0134] When an aggregation statement contains a grouping key, it indicates that the data needs to be divided into multiple groups based on a specific field or expression, and aggregation operations should be performed on each group separately. In this case, the results of partial aggregation can often directly satisfy the query requirements without needing to perform a final aggregation.

[0135] Step D2 above can be achieved by parsing the logical plan. Specifically, when parsing the logical plan, if the grouping expression of the aggregation node is not empty (i.e., a grouping key is defined), it can be confirmed that the goal of the aggregation operation here is to generate independent statistical results for each group. In this case, the results of partial aggregation are sufficient to reflect the statistical information of each group, so these partial results can be used directly without performing the final aggregation step. For example, in a use case of counting the number of employees by department, the number of employees in each department can be calculated independently through partial aggregation, without having to summarize the results of all departments to the central node. Similarly, in a use case of counting sales revenue by product category, the sales revenue of each category can be calculated on the corresponding partition or node and directly returned to the user.

[0136] Optionally, in addition to determining whether a grouping key exists in an aggregation statement through the grouping expression of the aggregation node, it is also possible to directly detect whether a grouping key exists in the aggregation statement. Directly detecting whether a grouping key exists in an aggregation statement may include the following steps S1 to S3:

[0137] S1. Parse the SQL query statement to extract information about aggregate functions and GROUP BY clauses.

[0138] Syntax parsing is the process of breaking down an SQL query statement into its constituent elements. By performing lexical and syntactic analysis on the query statement, the following key information can be extracted:

[0139] Aggregate functions, such as SUM, COUNT, AVG, MAX, and MIN, are used to perform statistical calculations.

[0140] The GROUP BY clause is used to specify the grouping field or expression, which determines how the data is divided into multiple groups.

[0141] Other relevant information includes filtering conditions (WHERE clause) and sorting rules (ORDER BY clause).

[0142] S2. If the query does not contain a GROUP BY clause, it is determined that the aggregation statement does not have a grouping key.

[0143] By examining the extracted information, it can be determined whether the query contains a GROUP BY clause. If the query does not define a GROUP BY clause, it is determined that the aggregation statement does not have a grouping key. In this case, the goal of the aggregation operation is usually to perform global statistics on the entire dataset.

[0144] S3. If the query contains a GROUP BY clause, then the aggregation statement is determined to have a grouping key.

[0145] If a GROUP BY clause is defined in the query, then the aggregation statement is considered to have a grouping key. In this case, the goal of the aggregation operation is to perform statistics on each group separately.

[0146] The following example illustrates this further. Let's consider a table named "employees" that stores employee salaries, with the structure shown in Table 1 below:

[0147] id name salary 1 Alice 5000 2 David 6000 3 Carol 7000

[0148] Table 1

[0149] Let's assume we need to calculate the total salary of all employees. The SQL query statement is as follows:

[0150] SELECT SUM(salary)AS total_salary FROM employees;

[0151] Syntax parsing of the above SQL query statement reveals the aggregate function SUM(salary). Furthermore, the absence of a GROUP BY clause indicates that the aggregate statement lacks a grouping key.

[0152] Continuing with the example above, let's assume we need to calculate the total salary for each department. The SQL query would look like this:

[0153]

[0154] Syntax parsing of the above SQL query reveals the aggregate function SUM(salary), and the query contains the definition of GROUP BY department_id. This indicates that the aggregate statement has a grouping key (department_id).

[0155] Therefore, it can be determined that no final aggregation is needed, and the subsequent execution steps can be:

[0156] Data is stored in partitions based on department_id. The total salary for each partition or node is calculated separately for its respective department (partial aggregation). The partial aggregation results for each department are returned directly, without performing the final aggregation process.

[0157] In the above implementation, the judgment is made directly based on the syntactic structure of the SQL statement, which avoids the complex process of generating and parsing logical plans and can quickly obtain results. Syntax parsing tools (such as lexical analyzers and parsers) already exist in most database systems, so these tools can be directly reused to implement the detection function. Since the generation of logical plans and the intervention of optimizers are not involved, computational overhead can be reduced.

[0158] Alternatively, in another embodiment, step 120 may specifically include the following steps E1 to E3:

[0159] E1. For each of the aggregate statements, obtain the subsequent execution statement corresponding to the aggregate statement.

[0160] In SQL queries, an aggregation statement may only be part of the overall query plan. To fully understand the impact of the aggregation operation on the entire query, we can obtain the subsequent execution statements corresponding to that aggregation statement. These subsequent execution statements may include, but are not limited to:

[0161] Other aggregation operations: such as nested aggregation functions or multiple aggregation steps.

[0162] Join operations: such as the JOIN clause, are used to merge data from different tables.

[0163] Filtering conditions: such as WHERE or HAVING clauses, are used to filter specific data records.

[0164] Sorting and grouping: such as the ORDER BY and GROUP BY clauses, used to organize the result set.

[0165] By obtaining and analyzing the subsequent execution statements mentioned above, we can better understand the position of the current aggregation operation in the entire query process and its impact on the final result.

[0166] Specifically, an SQL parser can be used to break down a query into its components, extracting all relevant aggregate statements. It then identifies how the output of each aggregate statement is used by subsequent statements. For example, is the result of one aggregate statement used as input to another, or is it used in a join operation? A dependency graph is then built to show the relationships between each aggregate statement and its subsequent execution statements. This helps to detect the flow of data and the order of processing.

[0167] E2. Perform semantic understanding on the subsequent executed statements to obtain the semantic understanding results of the subsequent executed statements.

[0168] The purpose of semantic understanding is to deeply analyze the actual meaning and function of subsequent execution statements to determine whether they will affect the requirements of aggregation operations. Specifically, it can help us understand whether the aforementioned subsequent statements will introduce new aggregation requirements or whether they will depend on existing aggregation results.

[0169] Specifically, semantic parsing is performed on each subsequent executed statement to identify its main function and role. For example, it checks whether there are further aggregation operations, join operations, or filtering conditions. The impact of subsequent statements on the current aggregation operation is evaluated in conjunction with the overall context of the query. For example, if a subsequent statement contains a global aggregation operation, the current aggregation statement may need to perform a final aggregation. It also checks whether subsequent statements depend on the result of the current aggregation statement. If subsequent statements require globally summarized information, the current aggregation statement typically needs to perform a final aggregation.

[0170] For example, let's suppose there exists an SQL query statement as follows:

[0171] SELECT department_id,AVG(salary)AS avg_salary

[0172] FROM employees

[0173] GROUP BY department_id

[0174] HAVING COUNT(*)>5;

[0175] Here, GROUP BY department_id means: group the data by department_id.

[0176] AVG (salary) means: calculate the average salary for each department (representing a local aggregation).

[0177] COUNT(*) means: count the number of employees in each department (representing another form of local aggregation).

[0178] HAVING COUNT(*)>5 means: filter the number of employees in each department and only keep departments with more than 5 employees.

[0179] The processing flow of the above SQL query statement is as follows:

[0180] First, group the data according to department_id; calculate AVG(salary) and COUNT(*) for each group; apply the condition HAVING COUNT(*)>5 to filter out groups that do not meet the condition; return the groups that meet the condition and their corresponding department_id and avg_salary.

[0181] In the SQL query above, AVG(salary) is a local aggregation operation, but HAVING COUNT(*)>5 is a subsequent execution statement that depends on the result of the global count. Therefore, it is necessary to ensure that the aggregation statement not only calculates the average salary for each department but also counts the number of employees in each department in order to apply the filtering conditions of the HAVING clause.

[0182] Specifically, the HAVING clause is used to filter the grouped results after an aggregation operation. HAVING COUNT(*)>5 explicitly requires filtering out departments with more than 5 employees. This means that the number of employees in each department needs to be accurately calculated, and the decision on whether to retain that department is based on that value.

[0183] E3. Based on the semantic understanding results, determine the aggregation state of the aggregation statement.

[0184] After completing the semantic understanding of subsequent executed statements, the aggregation state of the aggregation statement can be determined based on the understanding results. Specifically, the aggregation state can be divided into two cases:

[0185] If subsequent execution statements introduce new aggregation requirements or depend on the results of global summaries, then the current aggregation statement needs to perform a final aggregation; if subsequent execution statements only depend on the results of local aggregations and do not require information from global summaries, then the current aggregation statement can be determined not to perform a final aggregation.

[0186] Specifically, a comprehensive judgment is made by combining the characteristics of the aggregation statement itself (such as the presence of a grouping key) and the semantic understanding results of subsequent executed statements. Based on the judgment result, the aggregation status flag of the aggregation statement is updated so that appropriate optimization strategies can be applied during query execution.

[0187] Continuing with the SQL query example above, let's examine the following:

[0188] Aggregate statements: AVG(salary) and COUNT(*) are two aggregation operations.

[0189] The subsequent execution statement: HAVING COUNT(*)>5 requires the result of the global count.

[0190] Semantic understanding result: Since the HAVING clause depends on the result of the global count, the current aggregation statement needs to perform a final aggregation.

[0191] Specifically, in distributed databases or large-scale data processing environments, data is typically distributed across multiple nodes. To process data efficiently, partial aggregation might be performed on each node first, and then these partial results are passed to the central node for final aggregation. However, in this case, HAVING COUNT(*) > 5 introduces the problem that local counts are insufficient to meet the requirements.

[0192] Specifically, in a distributed environment, each node can independently calculate the number of employees in its local data (i.e., a partial count). However, these partial counts only reflect the data distribution of a particular node and cannot directly reflect the global count for the entire department. For example, suppose the data in the `employees` table is distributed across two nodes: Node 1 stores 3 records for department A, and Node 2 stores 4 records for department A. If relying solely on partial counts, Node 1 would consider the number of employees in department A to be 3, and Node 2 would consider the number to be 4. Neither node can independently determine whether the total number of employees in department A is greater than 5.

[0193] To correctly apply the condition HAVING COUNT(*)>5, the partial count results of all nodes need to be merged to obtain the global count for each department. Only after completing the global count can it be accurately determined which departments' employee counts meet the condition. In this case, the aggregation status of the aggregation statement can be marked as "final aggregation required".

[0194] The following section provides a detailed explanation of the typical execution process of the SQL query statement described above in a distributed environment:

[0195] Each node groups its local data and calculates the following two parts: the number of employees in each department (COUNT(*)) and the total salary and number of records in each department, which are used to calculate the average salary (AVG(salary)) later.

[0196] The partial aggregation results of each node (such as the local count of departments, the total salary, etc.) are passed to the central node.

[0197] At the central node: merge the partial counts of all nodes and calculate the global count (COUNT(*)) for each department. Merge the total salary and record count of all nodes and calculate the global average salary (AVG(salary)) for each department. Apply the condition HAVING COUNT(*)>5 to filter out departments that meet the criteria.

[0198] Return the departments that meet the criteria and their corresponding average salaries.

[0199] In the query example above, `HAVING COUNT(*)>5` explicitly requires a check of the global count for each department. Since local counts in a distributed environment cannot directly reflect the global situation, a final aggregation step is needed to merge the partial counts from all nodes to generate the global count. This makes the final aggregation an indispensable operation.

[0200] The following example illustrates steps E1 to E3.

[0201] Suppose there exists a table named 'orders' that stores order information, with the following structure:

[0202] id order_date customer amount 1 2023-01-01 Alice 100 2 2023-01-02 David 200 3 2023-01-01 Carol 300

[0203] Table 2

[0204] For Table 2 above, we need to calculate the total order amount for each day by date and return only those days where the total order amount is greater than 500. The SQL query statement is as follows:

[0205]

[0206]

[0207] GROUP BY DATE(order_date) means grouping the data by date.

[0208] SUM(amount) means: calculate the total order amount for each date (partial aggregation).

[0209] HAVING SUM(amount)>500 means: filter the total order amount for each date and only keep the dates with a total order amount greater than 500.

[0210] The processing flow of the above SQL query statement is as follows:

[0211] First, group the data according to DATE(order_date); calculate SUM(amount) for each group; apply the condition HAVING SUM(amount)>500 to filter out groups that do not meet the condition; return the groups that meet the condition and their corresponding order_day and total_amount.

[0212] In the SQL query above, HAVING SUM(amount)>500 is a condition for filtering the aggregation results of each group. Since SUM(amount) is calculated independently for each group, the value of SUM(amount) directly reflects the local aggregation result of that group, without depending on the global summary or the results of other groups.

[0213] Even in a distributed environment where data is distributed across multiple nodes, the SQL query above can still fulfill all requirements through partial aggregation, without requiring final aggregation. Here are the specific reasons:

[0214] Each node can independently group its local data and calculate the total order amount (SUM(amount)) for each group. Since the HAVING clause only needs to determine whether the SUM(amount) for each group is greater than 500, the results of local aggregation are sufficient for filtering. That is, the HAVING clause does not rely on global summary information across groups or nodes.

[0215] Since the HAVING clause only focuses on the local aggregation results of a single group, it can directly use the partial aggregation results of each node for filtering without passing these results to the central node for final aggregation.

[0216] Therefore, in the example above, the aggregation status of the aggregation statement can be directly marked as "no final aggregation required".

[0217] The following section provides a detailed explanation of the typical execution process of the SQL query statement described above in a distributed environment:

[0218] Each node groups its local data and calculates the following: grouped by DATE (order_date). Calculates the total order amount (SUM(amount)) for each group.

[0219] Each node directly applies the condition HAVING SUM(amount)>500 to the local aggregated results to filter out groups that meet the condition.

[0220] The filtered results from each node are returned directly to the user without the need for further merging of some results.

[0221] The above implementation improves the accuracy and efficiency of query optimization, making it particularly suitable for complex queries and distributed environments. This approach ensures query accuracy while minimizing unnecessary computational overhead, thereby enhancing overall performance.

[0222] 130. Obtain the original aggregation logic plan corresponding to the aggregation statement to be processed, wherein the original aggregation logic plan corresponds to an original aggregation logic plan, the original aggregation logic plan includes partial aggregation statements and final aggregation statements, the partial aggregation statements are used to perform partial aggregation operations, and the final aggregation statements are used to perform final aggregation operations.

[0223] The original aggregation logical plan is the original logical plan corresponding to the aggregation statement to be processed. The original aggregation logical plan consists of two main parts: partial aggregation and final aggregation.

[0224] The original aggregate physical plan is the physical plan corresponding to the original aggregate logical plan. In a database management system, the physical plan refers to the detailed steps and strategies for executing a query. The physical plan is generated based on the logical plan. The logical plan describes the operation flow and structure of the query, while the physical plan further refines these operations, specifying details such as specific algorithms, access paths, and execution order to ensure that the query can run efficiently on the underlying data storage system.

[0225] The original polymer scheme includes partial polymerization statements and final polymerization statements; please refer to [link / reference] for details. Figure 1c Some aggregation statements are responsible for performing local aggregation operations on individual nodes or partitions. The final aggregation statement is responsible for summing the results of all local aggregations into a global result.

[0226] 140. Replace the original aggregation logic plan with a partial aggregation logic plan so that when the partial aggregation logic plan is converted into a corresponding partial aggregation logic plan, the partial aggregation operation is executed, wherein the partial aggregation logic plan corresponds to the partial aggregation logic plan, and the partial aggregation logic plan includes the partial aggregation statement.

[0227] Some aggregation logic plans can retain only certain aggregation operations and remove unnecessary final aggregation steps. See details below. Figure 1d .

[0228] Optionally, in one implementation, the partial aggregation logic plan can be constructed by: keeping the data source information unchanged, that is, still using the same table or view as the data source; adjusting the aggregation operations, that is, retaining the partial aggregation operations and no longer retaining the final aggregation operations; and adjusting other operations (such as joins, filters, etc.) according to the actual situation to ensure that they match the new partial aggregation logic plan.

[0229] Because the partial aggregation logical plan includes the partial aggregation statements while removing the final aggregation statement, when the partial aggregation logical plan is converted into the corresponding partial aggregation logical plan, only the necessary local aggregation operations can be performed. For example: partial aggregation statements.

[0230] In the above implementation, the original aggregation logic plan is replaced with a partial aggregation logic plan, so that the aggregation statement to be processed performs a partial aggregation operation, instead of performing both a partial aggregation operation and a final aggregation operation.

[0231] Optionally, in one specific implementation, steps 110 to 140 can be implemented using an optimization rule called `useLogicalPartialAggregate Rule`. This rule, `useLogicalPartialAggregate Rule`, is designed to follow Spark's pluggable architecture. Users can enable or disable this optimization feature through simple configuration without modifying the underlying code. Optionally, users can choose whether to enable this optimization rule based on their actual needs. If this feature is required, simply add the corresponding parameters to the Spark configuration file. For example:

[0232] spark.sql.optimizer.useLogicalPartialAggregate=true

[0233] In Spark's Catalyst optimizer, the `useLogicalPartialAggregate` rule performs in-depth analysis of the logical plan of SQL queries, identifying scenarios where only partial aggregation results can be relied upon. For queries that meet the criteria, this rule replaces the original logical plan (including partial and final aggregations) with a partial aggregation logical plan containing only partial aggregations. By removing the final aggregation step, the need for data transfer across nodes is reduced, significantly lowering network bandwidth and computational overhead.

[0234] Alternatively, in one implementation, please refer to [link to relevant documentation]. Figure 1e , Figure 1e A schematic flowchart illustrating the SQL query statement is shown, wherein:

[0235] SQL Query: This is the SQL query statement submitted by the user. It is the starting point of the entire query processing flow, where the user specifies the operation to be performed through the SQL statement.

[0236] Dataset: Represents the data source for which a query operation needs to be performed, typically data stored in a distributed file system or other data storage system.

[0237] Unresolved Logical Plan: This refers to the preliminary logical plan, which contains information such as table names and column names that have not yet been resolved.

[0238] Catalog: This represents the location where metadata information is stored, including table structure, indexes, and other information, used to resolve unresolved logical plans.

[0239] Logical Plan: This refers to the parsed logical plan, where table names, column names, and other information have been parsed.

[0240] Optimized Logical Plan: This refers to a logically optimized plan that incorporates more efficient query execution strategies.

[0241] Physical Plans: Multiple possible physical execution plans, each with a different execution strategy.

[0242] Cost Model: Used to evaluate the cost of different physical plans and select the physical plan with the lowest cost as the final execution plan.

[0243] Selected Physical Plan: The optimal physical execution plan selected based on the cost model.

[0244] Resilient Distributed Datasets (RDDs): The basic data abstraction in Spark, representing an immutable, partitioned collection of records that can be stored in parallel across a cluster.

[0245] Analysis phase: In this phase, Spark performs syntax analysis and type checking on the SQL query, generating an unresolved logical plan.

[0246] Logical Optimization Phase: In this phase, the logical plan is optimized to generate an optimized logical plan. The insertion of the `useLogicalPartialAggregate` rule occurs during this phase.

[0247] Physical Planning Phase: In this phase, the optimized logical plan is converted into specific physical execution plans, and the optimal execution strategy is selected.

[0248] Code Generation Phase: The selected physical plan is converted into specific executable code.

[0249] Optionally, the `useLogicalPartialAggregate` rule can be inserted during the logic optimization phase to optimize SQL queries involving multi-table joins and aggregation operations. The specific steps are as follows:

[0250] In the logical plan optimization layer of the in-memory distributed computing engine, a logical plan rule, `useLogicalPartialAggregate`, is implemented. When analyzing SQL statements with partial aggregations using a GROUP BY key, the `useLogicalPartialAggregate` optimization rule can be directly applied. Then, partial aggregations are used directly at the physical plan layer, reducing final aggregations and achieving performance optimization.

[0251] In the above implementation, through these steps, Spark can handle complex multi-table joins and aggregation queries more efficiently, significantly improving the overall execution efficiency of the job.

[0252] The following section will illustrate the effect of the `useLogicalPartialAggregate` optimization rule by comparing the physical plans before and after optimization in the example:

[0253] Let's assume the original physical plan is as follows:

[0254]

[0255] Analysis of the physical plan before optimization shows that:

[0256] The first step is to perform a partial aggregation operation to calculate the local count result within each partition. The HashAggregate operation is used, specifying keys = [ss_customer_sk#120L] and functions = [partial_count(1)].

[0257] After partial aggregation is complete, the results need to be redistributed to different partitions via Exchange operations. Data is hash-partitioned using hashpartitioning(ss_customer_sk#120L,250) to ensure that data with the same ss_customer_sk is assigned to the same partition.

[0258] After data redistribution, HashAggregate is used again to perform the final aggregation operation, generating a global summary result. The function `functions = [count(1)]` is used to indicate that all local count results are merged.

[0259] The Exchange operation resulted in a large amount of data transfer across nodes, increasing network bandwidth consumption. The final aggregation operation required summarizing all local aggregation results at the central node, increasing the computational burden. Furthermore, the storage and transfer of intermediate results significantly increased disk I / O and memory usage.

[0260] The optimized physics plan is as follows:

[0261]

[0262] Analysis of the optimized physics plan above shows that:

[0263] Similar to the previous method, partial aggregation is still performed to calculate the local count results within each partition. The HashAggregate operation is used, specifying keys = [ss_customer_sk#120L] and functions = [partial_count(1)].

[0264] The optimized physical plan completely removes Exchange operations, avoiding cross-node data transfer. The optimized physical plan directly returns the results of partial aggregations without performing the final aggregation operation. This indicates that the results of partial aggregations are sufficient to meet query requirements, eliminating the need for further global summarization.

[0265] A comparison of the physical plans before and after optimization reveals that, compared to the original physical plan, the optimized plan significantly reduces cross-node data transfer requirements by eliminating Exchange operations, thereby reducing network bandwidth usage; it avoids the final aggregation operation, reducing the computational load on the central node; and it significantly reduces disk I / O and memory usage because it does not require storing and transmitting intermediate results. The overall execution time is shortened, especially in scenarios with large-scale datasets and complex queries, where the optimization effect is particularly significant.

[0266] As can be seen from the above, the core idea of ​​optimization is to identify scenarios that can rely only on partial aggregation results by analyzing the logical plan, and then directly remove the final aggregation operation in the physical plan.

[0267] Specifically:

[0268] During the logic optimization phase, the `useLogicalPartialAggregate` rule is inserted to analyze whether the SQL query contains a `GROUP BY` clause and aggregate functions. If it is found that the local aggregation results can already meet the query requirements, only some aggregation operations are retained in the physical plan, and unnecessary Exchange and final aggregation operations are removed.

[0269] By comparing the physical plans before and after optimization, it can be seen that the main effects of the `useLogicalPartialAggregate` optimization rule are: reducing unnecessary data redistribution (exchange) operations and lowering network transmission overhead; removing the final aggregation operation, reducing computational burden and resource consumption; and improving the overall execution efficiency of queries, especially in scenarios involving large-scale datasets and complex queries. This optimization strategy is not only applicable to Spark but can also be extended to other distributed computing engines or database systems to provide users with more efficient data processing capabilities.

[0270] In the data processing method provided in this application embodiment, a structured query statement is obtained, which includes multiple aggregation statements. From these multiple aggregation statements, aggregation statements that do not require final aggregation (denoted as aggregate statements to be processed) are identified. The original aggregation logic plan of the aggregate statements to be processed is obtained, wherein the original aggregation logic plan corresponds to the original aggregation logic plan, which is used to perform partial aggregation operations and the final aggregation operation. The original aggregation logic plan is replaced with the partial aggregation logic plan so that when the partial aggregation logic plan is converted into the corresponding partial aggregation logic plan, partial aggregation operations are performed, instead of the partial aggregation operations and the final aggregation operations corresponding to the original aggregation logic plan. In this application embodiment, aggregate statements to be processed that do not require final aggregation are first identified from multiple aggregation statements. Then, the original aggregation logic plan corresponding to the aggregate statements to be processed is replaced with the partial aggregation logic plan so that the aggregate statements to be processed perform partial aggregation operations, instead of performing both partial and final aggregation operations. By identifying aggregate statements that do not require final aggregation and replacing their corresponding logic plans, this application embodiment can avoid the execution of meaningless and computationally wasteful final aggregation operations.

[0271] The embodiments of this application can save computing resources.

[0272] In this embodiment, the method of this application embodiment will be described in detail. The data processing method provided in this application embodiment can be executed by an electronic device. In this embodiment, the electronic device is a server as an example for description, such as... Figure 2 As shown, the specific process of a data processing method is as follows:

[0273] 201. Obtain a structured query statement, wherein the structured query statement includes multiple aggregate statements.

[0274] 202. If the aggregation statement has a grouping key, the aggregation state of the aggregation statement is determined to be that no final aggregation is required, wherein the aggregation statement that does not require final aggregation is an aggregation statement to be processed.

[0275] Optionally, in addition to determining the aggregation state of the aggregation statement through step 202 above, the aggregation state of the aggregation statement can also be determined in other ways, such as:

[0276] For each aggregation statement, obtain the original aggregation logic plan corresponding to the aggregation statement; based on the original aggregation logic plan, determine the aggregation state of the corresponding aggregation statement, wherein the aggregation state includes those that require final aggregation and those that do not require final aggregation.

[0277] Optionally, in one implementation, the original aggregation logic plan includes aggregation nodes; based on the original aggregation logic plan, determining the aggregation state of the corresponding aggregation statement includes:

[0278] Based on the grouping expression of the aggregation node, determine the grouping key state of the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs; based on the grouping key state of the aggregation statement, determine the aggregation state of the aggregation statement.

[0279] Optionally, the grouping key status includes the absence of a grouping key; correspondingly, the above steps may include: if the grouping expression of the aggregation node is empty, then determining that the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs does not have a grouping key; if the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs does not have a grouping key, determining that the aggregation status of the aggregation statement is that final aggregation is required.

[0280] Optionally, the grouping key status includes the existence of a grouping key; correspondingly, the above steps may include: if the grouping expression of the aggregation node is not empty, then determining that the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs has a grouping key; if the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs has a grouping key, determining that the aggregation status of the aggregation statement is that no final aggregation is required.

[0281] Optionally, in another implementation, determining the aggregation state of the corresponding aggregation statement based on the original aggregation logic plan includes: checking the pre-partitioning state of the data source in the original aggregation logic plan, wherein the data source is data to be queried by the structured query statement; and determining the aggregation state of the aggregation statement based on the pre-partitioning state of the data source.

[0282] Optionally, the pre-partitioning status includes pre-partitioned and non-pre-partitioned; correspondingly, checking the pre-partitioning status of the data source in the original aggregation logic plan includes:

[0283] If the data in the data source is partitioned and stored according to a grouping key, then the pre-partitioning status is determined to be pre-partitioned; if the data in the data source is not partitioned and stored according to a grouping key, then the pre-partitioning status is determined to be pre-partitioned.

[0284] Accordingly, determining the aggregation state of the aggregation statement based on the pre-partitioning state of the data source includes:

[0285] If the pre-partitioning state is one in which pre-partitioning has been performed, the aggregation state is determined to be one in which final aggregation is not required; if the pre-partitioning state is one in which pre-partitioning has not been performed, the aggregation state is determined to be one in which final aggregation is required.

[0286] Optionally, in addition to determining the aggregation state of the aggregation statement through step 202 above, the aggregation state of the aggregation statement can also be determined through other methods, such as:

[0287] For each aggregation statement, obtain the subsequent execution statement corresponding to the aggregation statement; perform semantic understanding on the subsequent execution statement to obtain the semantic understanding result of the subsequent execution statement; and determine the aggregation state of the aggregation statement based on the semantic understanding result.

[0288] It should be understood that the specific method for determining the aggregation state of an aggregation statement should not be construed as a limitation of this application.

[0289] 203. Obtain the original aggregation logic plan corresponding to the aggregation statement to be processed, wherein the original aggregation logic plan corresponds to an original aggregation logic plan, the original aggregation logic plan includes partial aggregation statements and final aggregation statements, the partial aggregation statements are used to perform partial aggregation operations, and the final aggregation statements are used to perform final aggregation operations.

[0290] 204. Replace the original aggregation logic plan with a partial aggregation logic plan so that when the partial aggregation logic plan is converted into a corresponding partial aggregation logic plan, the partial aggregation operation is executed, wherein the partial aggregation logic plan corresponds to the partial aggregation logic plan, and the partial aggregation logic plan includes the partial aggregation statement.

[0291] The specific execution process of steps 201 to 204 has been explained in detail above, and will not be repeated here.

[0292] In the data processing method provided in this application embodiment, a structured query statement is obtained, which includes multiple aggregation statements. From these multiple aggregation statements, aggregation statements that do not require final aggregation (denoted as aggregate statements to be processed) are identified. The original aggregation logic plan of the aggregate statements to be processed is obtained, wherein the original aggregation logic plan corresponds to the original aggregation logic plan, which is used to perform partial aggregation operations and the final aggregation operation. The original aggregation logic plan is replaced with the partial aggregation logic plan so that when the partial aggregation logic plan is converted into the corresponding partial aggregation logic plan, partial aggregation operations are performed, instead of the partial aggregation operations and the final aggregation operations corresponding to the original aggregation logic plan. In this application embodiment, aggregate statements to be processed that do not require final aggregation are first identified from multiple aggregation statements. Then, the original aggregation logic plan corresponding to the aggregate statements to be processed is replaced with the partial aggregation logic plan so that the aggregate statements to be processed perform partial aggregation operations, instead of performing both partial and final aggregation operations. By identifying aggregate statements that do not require final aggregation and replacing their corresponding logic plans, this application embodiment can avoid the execution of meaningless and computationally wasteful final aggregation operations.

[0293] The embodiments of this application can save computing resources.

[0294] This application's embodiments significantly optimize the performance of the distributed computing engine by reducing the generation of Physical Final Aggregate operations at the physical plan execution layer. Specifically, the core of this optimization strategy lies in avoiding unnecessary global aggregation steps, thereby reducing the generation of Shuffle operations in the distributed computing engine. Shuffle refers to the process where data is transferred from one node to another, or moved from one partition to another, when an operation requires redistributing data (e.g., grouping or merging data by a key). The Shuffle process involves data sorting, grouping, and repartitioning, all of which incur significant overhead. The aforementioned process reduces the generation of Shuffle operations, thereby reducing computational power consumption and improving efficiency.

[0295] This application's embodiments reduce the computational load during the physical aggregation process, thereby lowering the computational load and network bandwidth requirements, and improving the overall performance of the execution process. The reduced computational load also lowers resource consumption. This, in turn, improves the computational performance of the distributed computing engine and significantly shortens job execution time.

[0296] This application's embodiments achieve powerful SQL logical plan optimization capabilities through in-depth analysis of the SQL query's logical plan and combined with optimization strategies for aggregation operations. This optimization not only significantly reduces the data computation density in the distributed computing engine but also lowers system resource consumption at multiple levels, thereby comprehensively improving the overall execution efficiency of the job. Furthermore, the core idea of ​​this application's embodiments—"reducing unnecessary computation and data transmission through logical plan optimization"—has strong versatility and can be widely applied to various database architectures and computing engines, such as:

[0297] MPP architecture: In the Massively Parallel Processing (MPP) architecture, the embodiments of this application can effectively reduce the communication overhead between nodes and improve the efficiency of distributed computing.

[0298] Modern data warehouses: Whether it is a cloud-based distributed data warehouse (such as Snowflake, BigQuery) or a locally deployed data warehouse (such as Greenplum, ClickHouse), the optimization ideas of the embodiments in this application can be used as a reference.

[0299] Traditional databases: Even traditional relational databases on a single machine (such as MySQL and PostgreSQL) can reduce computational and I / O overhead and improve query performance through similar optimization strategies.

[0300] To better implement the above methods, embodiments of this application also provide a data processing apparatus, such as... Figure 3 As shown, the device includes:

[0301] The statement acquisition unit 301 is used to acquire a structured query statement, wherein the structured query statement includes multiple aggregate statements;

[0302] The statement recognition unit 302 is used to identify from the plurality of aggregation statements that do not require final aggregation, wherein the aggregation statements that do not require final aggregation are aggregation statements to be processed.

[0303] The original logic plan unit 303 is used to obtain the original aggregation logic plan corresponding to the aggregation statement to be processed. The original aggregation logic plan corresponds to an original aggregation processing plan, which includes a partial aggregation statement and a final aggregation statement. The partial aggregation statement is used to perform partial aggregation operations, and the final aggregation statement is used to perform final aggregation operations.

[0304] The logic plan replacement unit 304 is used to replace the original aggregation logic plan with a partial aggregation logic plan so that when the partial aggregation logic plan is converted into a corresponding partial aggregation logic plan, the partial aggregation operation is executed. The partial aggregation logic plan corresponds to the partial aggregation logic plan, and the partial aggregation logic plan includes the partial aggregation statement.

[0305] In one embodiment, the statement recognition unit 302 includes:

[0306] The logic plan subunit is used to obtain the original aggregation logic plan corresponding to each aggregation statement;

[0307] The aggregation state subunit is used to determine the aggregation state of the corresponding aggregation statement based on the original aggregation logic plan, wherein the aggregation state includes whether final aggregation is required or not.

[0308] In one implementation, the original aggregation logic plan includes aggregation nodes; correspondingly, the aggregation state subunit includes:

[0309] The grouping key state sub-unit is used to determine the grouping key state of the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs, based on the grouping expression of the aggregation node.

[0310] The aggregation state sub-unit is used to determine the aggregation state of the aggregation statement based on the grouping key state of the aggregation statement.

[0311] In one implementation, the group key state includes the absence of a group key;

[0312] Correspondingly, the group key state sub-unit is specifically used to: determine that when the group expression of the aggregation node is empty, the aggregation statement corresponding to the original aggregation logic plan to which the group expression belongs does not have a group key;

[0313] The aggregation state sub-unit is specifically used to: determine that the aggregation state of the aggregation statement needs to be finally aggregated when the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs does not have a grouping key.

[0314] In one implementation, the grouping key state includes the presence of a grouping key;

[0315] Correspondingly, the group key state sub-unit is specifically used to: determine that the aggregation statement corresponding to the original aggregation logic plan to which the group expression belongs has a group key when the group expression of the aggregation node is not empty;

[0316] The aggregation state sub-unit is specifically used to: determine that the aggregation state of the aggregation statement does not require final aggregation when the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs has a grouping key.

[0317] In one implementation, the aggregation state subunit includes:

[0318] The pre-partitioning sub-unit is used to check the pre-partitioning status of the data source in the original aggregation logic plan, wherein the data source is the data to be queried by the structured query statement;

[0319] The aggregation state sub-unit is used to determine the aggregation state of the aggregation statement based on the pre-partitioning state of the data source.

[0320] In one implementation, the pre-partitioning state includes pre-partitioning performed and pre-partitioning not performed;

[0321] The pre-partitioning sub-unit is specifically used to: determine that the pre-partitioning status has been pre-partitioned when the data in the data source is partitioned and stored according to the grouping key;

[0322] When the data in the data source is not partitioned and stored according to the grouping key, the pre-partitioning state is determined to be no pre-partitioning.

[0323] In one implementation, the aggregated state sub-unit is specifically used for:

[0324] If the pre-partitioning state is one in which pre-partitioning has already been performed, then the aggregation state is determined to be one in which final aggregation is not required.

[0325] If the pre-partitioning state is that no pre-partitioning has been performed, the aggregation state is determined to be that final aggregation is required.

[0326] In one embodiment, the statement recognition unit 302 includes:

[0327] The first determination subunit is used to determine that the aggregation state of the aggregation statement needs to be finalized when the aggregation statement does not have a grouping key.

[0328] The second determination subunit is used to determine that the aggregation state of the aggregation statement does not require final aggregation when the aggregation statement has a grouping key.

[0329] In one embodiment, the statement recognition unit 302 includes:

[0330] The subsequent execution subunit is used to obtain the subsequent execution statement corresponding to each of the aggregate statements;

[0331] A semantic understanding subunit is used to perform semantic understanding on the subsequent execution statements to obtain the semantic understanding result of the subsequent execution statements;

[0332] The aggregation state subunit is used to determine the aggregation state of the aggregation statement based on the semantic understanding result.

[0333] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0334] In the data processing method provided in this application embodiment, a structured query statement is obtained, which includes multiple aggregation statements. From these multiple aggregation statements, aggregation statements that do not require final aggregation (denoted as aggregate statements to be processed) are identified. The original aggregation logic plan of the aggregate statements to be processed is obtained, wherein the original aggregation logic plan corresponds to the original aggregation logic plan, which is used to perform partial aggregation operations and the final aggregation operation. The original aggregation logic plan is replaced with the partial aggregation logic plan so that when the partial aggregation logic plan is converted into the corresponding partial aggregation logic plan, partial aggregation operations are performed, instead of the partial aggregation operations and the final aggregation operations corresponding to the original aggregation logic plan. In this application embodiment, aggregate statements to be processed that do not require final aggregation are first identified from multiple aggregation statements. Then, the original aggregation logic plan corresponding to the aggregate statements to be processed is replaced with the partial aggregation logic plan so that the aggregate statements to be processed perform partial aggregation operations, instead of performing both partial and final aggregation operations. By identifying aggregate statements that do not require final aggregation and replacing their corresponding logic plans, this application embodiment can avoid the execution of meaningless and computationally wasteful final aggregation operations.

[0335] The embodiments of this application can save computing resources.

[0336] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0337] In some embodiments, the data processing apparatus may also be integrated into multiple electronic devices, such as multiple servers, with the data processing method of this application being implemented by the multiple servers.

[0338] In this embodiment, a server will be used as an example for detailed description. For example, ... Figure 4 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0339] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0340] The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402. In some embodiments, the processor 401 may include one or more processing cores; in some embodiments, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 401.

[0341] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing model training by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0342] The electronic device also includes a power supply 403 that supplies power to the various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0343] The electronic device may also include an input module 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0344] The electronic device may also include a communication module 405. In some embodiments, the communication module 405 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 405 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0345] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402, thereby realizing the various functions in the various method embodiments of this application, as follows:

[0346] Obtain a structured query statement, wherein the structured query statement includes multiple aggregation statements; identify aggregation statements that do not require final aggregation from the multiple aggregation statements, wherein the aggregation statements that do not require final aggregation are aggregate statements to be processed; obtain the original aggregation logic plan corresponding to the aggregate statements to be processed, wherein the original aggregation logic plan corresponds to an original aggregation management plan, wherein the original aggregation management plan includes partial aggregation statements and a final aggregation statement, wherein the partial aggregation statements are used to perform partial aggregation operations, and the final aggregation statements are used to perform final aggregation operations; replace the original aggregation logic plan with the partial aggregation logic plan, so that when the partial aggregation logic plan is converted into the corresponding partial aggregation management plan, the partial aggregation operation is executed, wherein the partial aggregation logic plan corresponds to the partial aggregation management plan, and the partial aggregation management plan includes the partial aggregation statements.

[0347] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0348] Therefore, embodiments of this application provide a computer-readable storage medium storing instructions that can be loaded by a processor to perform steps in any of the data processing methods provided in embodiments of this application. For example, the instructions can perform the following steps:

[0349] Obtain a structured query statement, wherein the structured query statement includes multiple aggregation statements; identify aggregation statements that do not require final aggregation from the multiple aggregation statements, wherein the aggregation statements that do not require final aggregation are aggregate statements to be processed; obtain the original aggregation logic plan corresponding to the aggregate statements to be processed, wherein the original aggregation logic plan corresponds to an original aggregation management plan, wherein the original aggregation management plan includes partial aggregation statements and a final aggregation statement, wherein the partial aggregation statements are used to perform partial aggregation operations, and the final aggregation statements are used to perform final aggregation operations; replace the original aggregation logic plan with the partial aggregation logic plan, so that when the partial aggregation logic plan is converted into the corresponding partial aggregation management plan, the partial aggregation operation is executed, wherein the partial aggregation logic plan corresponds to the partial aggregation management plan, and the partial aggregation management plan includes the partial aggregation statements.

[0350] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0351] According to one aspect of this application, a computer program product or computer program is provided, comprising instructions stored in a computer-readable storage medium. A processor of a computer device reads the instructions from the computer-readable storage medium and executes the instructions, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.

[0352] Since the instructions stored in the storage medium can execute the steps of any of the data processing methods provided in the embodiments of this application, the beneficial effects that any of the data processing methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0353] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0354] The foregoing has provided a detailed description of a data processing method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain a structured query statement, wherein the structured query statement includes multiple aggregate statements; From the plurality of aggregation statements, identify the aggregation statements that do not require final aggregation, wherein the aggregation statements that do not require final aggregation are the aggregation statements to be processed; Obtain the original aggregation logic plan corresponding to the aggregation statement to be processed, wherein the original aggregation logic plan corresponds to an original aggregation processing plan, the original aggregation processing plan includes partial aggregation statements and final aggregation statements, the partial aggregation statements are used to perform partial aggregation operations, and the final aggregation statements are used to perform final aggregation operations; The original aggregation logic plan is replaced with a partial aggregation logic plan so that when the partial aggregation logic plan is converted into a corresponding partial aggregation logic plan, the partial aggregation operation is executed. The partial aggregation logic plan corresponds to the partial aggregation logic plan, and the partial aggregation logic plan includes the partial aggregation statement.

2. The method as described in claim 1, characterized in that, The step of identifying aggregation statements from the plurality of aggregation statements that do not require final aggregation includes: For each of the aggregation statements, obtain the original aggregation logic plan corresponding to the aggregation statement; Based on the original aggregation logic plan, the aggregation state of the corresponding aggregation statement is determined, wherein the aggregation state includes those requiring final aggregation and those not requiring final aggregation.

3. The method as described in claim 2, characterized in that, The original aggregation logic plan includes aggregation nodes; The process of determining the aggregation state of the corresponding aggregation statement based on the original aggregation logic plan includes: Based on the grouping expression of the aggregation node, determine the grouping key state of the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs; The aggregation state of the aggregation statement is determined based on the grouping key state of the aggregation statement.

4. The method as described in claim 3, characterized in that, The group key status includes the absence of a group key; The determination of the grouping key state of the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs, based on the grouping expression of the aggregation node, includes: If the grouping expression of the aggregation node is empty, it is determined that the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs does not have a grouping key; Determining the aggregation state of the aggregation statement based on the grouping key state of the aggregation statement includes: If the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs does not have a grouping key, the aggregation state of the aggregation statement is determined to be that final aggregation is required.

5. The method as described in claim 3, characterized in that, The grouping key status includes the presence of a grouping key; The determination of the grouping key state of the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs, based on the grouping expression of the aggregation node, includes: If the grouping expression of the aggregation node is not empty, then it is determined that the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs has a grouping key; Determining the aggregation state of the aggregation statement based on the grouping key state of the aggregation statement includes: If the aggregation statement corresponding to the original aggregation logic plan to which the grouping expression belongs has a grouping key, the aggregation state of the aggregation statement is determined to be that no final aggregation is required.

6. The method as described in claim 2, characterized in that, The process of determining the aggregation state of the corresponding aggregation statement based on the original aggregation logic plan includes: In the original aggregation logic plan, the pre-partition status of the data source is checked, wherein the data source is the data to be queried by the structured query statement; The aggregation state of the aggregation statement is determined based on the pre-partitioning state of the data source.

7. The method as described in claim 6, characterized in that, The pre-partitioning status includes pre-partitioning performed and pre-partitioning not performed; The step of checking the pre-partitioning status of the data source in the original aggregation logic plan includes: If the data in the data source is partitioned and stored according to the grouping key, then the pre-partitioning state is determined to be pre-partitioned; If the data in the data source is not partitioned and stored according to the grouping key, then the pre-partitioning status is determined to be that no pre-partitioning has been performed.

8. The method as described in claim 6, characterized in that, Determining the aggregation state of the aggregation statement based on the pre-partitioning state of the data source includes: If the pre-partitioning state is one in which pre-partitioning has already been performed, then the aggregation state is determined to be one in which final aggregation is not required. If the pre-partitioning state is that no pre-partitioning has been performed, the aggregation state is determined to be that final aggregation is required.

9. The method as described in claim 1, characterized in that, The step of identifying aggregation statements from the plurality of aggregation statements that do not require final aggregation includes: If the aggregation statement does not have a grouping key, the aggregation state of the aggregation statement is determined to be that final aggregation is required; If the aggregation statement has a grouping key, the aggregation state of the aggregation statement is determined to be that no final aggregation is required.

10. The method as described in claim 1, characterized in that, The step of identifying aggregation statements from the plurality of aggregation statements that do not require final aggregation includes: For each of the aggregate statements, obtain the subsequent execution statement corresponding to the aggregate statement; Semantic understanding is performed on the subsequent executed statements to obtain the semantic understanding results of the subsequent executed statements; Based on the semantic understanding results, the aggregation state of the aggregation statement is determined.

11. A data processing apparatus, characterized in that, The device includes: The statement acquisition unit is used to acquire structured query statements, wherein the structured query statements include multiple aggregate statements; The statement recognition unit is used to identify, from the plurality of aggregation statements, the aggregation statements that do not need to be finally aggregated, wherein the aggregation statements that do not need to be finally aggregated are the aggregation statements to be processed; The original logic plan unit is used to obtain the original aggregation logic plan corresponding to the aggregation statement to be processed. The original aggregation logic plan corresponds to an original aggregation processing plan, which includes partial aggregation statements and final aggregation statements. The partial aggregation statements are used to perform partial aggregation operations, and the final aggregation statements are used to perform final aggregation operations. A logic plan replacement unit is used to replace the original aggregation logic plan with a partial aggregation logic plan, so that when the partial aggregation logic plan is converted into a corresponding partial aggregation logic plan, the partial aggregation operation is executed, wherein the partial aggregation logic plan corresponds to the partial aggregation logic plan, and the partial aggregation logic plan includes the partial aggregation statement.

12. An electronic device, characterized in that, It includes a processor and a memory, the memory storing instructions; the processor loads instructions from the memory to perform the steps of the data processing method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions adapted for loading by a processor to perform the steps of the data processing method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, It includes instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 10.

Images