Data processing method, device, equipment, readable storage medium and program product

Abstract

Embodiments of the present application provide a data processing method, device, equipment, readable storage medium and program product. The method comprises: determining a physical shuffle node from a physical operator tree corresponding to a database task; obtaining a first partition quantity recorded by the physical shuffle node, and obtaining a second partition quantity recorded by a child node of the physical shuffle node; if the first partition quantity is equal to a target value and the second partition quantity is equal to the target value, performing optimization processing on the physical operator tree, the optimization processing comprising: deleting a shuffle operation corresponding to the physical shuffle node in the physical operator tree; and responding to the database task according to the optimized physical operator tree. Through the embodiments of the present application, the Shuffle data quantity can be reduced, and the computing performance of the distributed computing engine can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to data processing methods, data processing apparatus, computer equipment, computer-readable storage media, and computer program products. Background Technology

[0002] In big data analytics and processing, data shuffling is one of the core operations in distributed computing and also one of the performance bottlenecks. Essentially, it involves repartitioning and recombining data across various computing nodes. During data shuffling, a large amount of data moves between different nodes, significantly increasing network bandwidth pressure and computational latency. This is especially true in shuff-intensive scenarios, where the computational performance of the distributed computing engine will be severely impacted. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, readable storage medium, and program product that can reduce the amount of shuffle data and improve the computing performance of a distributed computing engine.

[0004] On one hand, embodiments of this application provide a data processing method, the method comprising:

[0005] Determine the physical shuffling nodes from the physical operator tree corresponding to the database task;

[0006] Get the number of the first partition recorded by the physical shuffle node, and get the number of the second partition recorded by the child nodes of the physical shuffle node;

[0007] If the number of the first partition is equal to the target value and the number of the second partition is equal to the target value, then the physical operator tree is optimized. The optimization process includes deleting the shuffle operation corresponding to the physical shuffle node in the physical operator tree.

[0008] The database task is responded to according to the optimized physical operator tree.

[0009] On the other hand, embodiments of this application provide a data processing apparatus, which includes:

[0010] The acquisition unit is used to determine the physical shuffling node from the physical operator tree corresponding to the database task;

[0011] The acquisition unit is also used to acquire the number of first partitions recorded by the physical shuffling node and the number of second partitions recorded by the child nodes of the physical shuffling node.

[0012] The execution unit is used to perform optimization processing on the physical operator tree if the number of the first partition is equal to the target value and the number of the second partition is equal to the target value; the optimization processing includes: deleting the shuffling operation corresponding to the physical shuffling node in the physical operator tree;

[0013] The response unit is used to respond to database tasks according to the optimized physical operator tree.

[0014] In one possible implementation, the aforementioned execution unit, when performing optimization processing on the physical operator tree, is specifically used for:

[0015] Determine the parent node of the data shuffling node from the physical operator tree;

[0016] Delete the data shuffling node in the physical operator tree, as well as the node connections between the data shuffling node and its parent and child nodes.

[0017] Establish node connections between parent and child nodes.

[0018] In one possible implementation, the above execution unit is also used for:

[0019] When the number of the first partition is equal to the target value and the number of the second partition is equal to the target value, obtain the shuffling field of the physical shuffling node and the partition field indicated by the database partitioning rule corresponding to the database task;

[0020] If the shuffle field matches the partition field, an optimization step is triggered on the physical operator tree.

[0021] In one possible implementation, the execution unit described above, when used to trigger optimization processing on the physical operator tree if the shuffle field matches the partition field, is specifically used for:

[0022] If the shuffle field matches the partition field, then obtain the first data volume of the data source corresponding to the child node of the physical shuffle node;

[0023] When the first data volume is less than or equal to the first data volume threshold, the step of performing optimization processing on the physical operator tree is triggered.

[0024] In one possible implementation, before determining the physical shuffling node from the physical operator tree corresponding to the database task, the aforementioned execution unit is further configured to:

[0025] Determine the logical shuffling nodes in the logical operator tree corresponding to the database task;

[0026] Get the number of third partitions recorded by the logical shuffle node, and get the number of fourth partitions recorded by the child nodes of the logical shuffle node.

[0027] When the number of at least one of the third and fourth partitions is not equal to the target value, obtain the second data volume of the data source corresponding to the child node of the logical shuffle node;

[0028] If the second data volume is less than or equal to the second data volume threshold, then adjust the number of partitions in the third and fourth partitions that are not equal to the target value to the target value.

[0029] Generate the physical operator tree corresponding to the database task based on the adjusted logical operator tree.

[0030] In one possible implementation, the above execution unit is also used for:

[0031] Real-time acquisition of system operation data from the distributed computing engine;

[0032] The target data volume threshold is dynamically adjusted based on system operation data; whereby system operation data includes one or more of available memory, CPU utilization, and network load, and the target data volume threshold includes one or more of a first data volume threshold and a second data volume threshold.

[0033] In one possible implementation, the above execution unit is also used for:

[0034] During the processing phase of the physical operator optimization layer of the distributed computing engine, if the physical operator optimization layer is configured with shuffling optimization rules, the step of determining the physical shuffling node from the physical operator tree corresponding to the database task is triggered.

[0035] Specifically, when performing optimization processing on the physical operator tree, the aforementioned execution unit is used for:

[0036] The physical operator tree is optimized according to the shuffling optimization rules configured in the physical operator optimization layer.

[0037] In one possible implementation, having the number of the first partition equal to the target value and the number of the second partition equal to the target value means that both the number of the first partition and the number of the second partition are equal to 1.

[0038] Accordingly, embodiments of this application provide a computer device, which includes:

[0039] A processor is a tool for implementing computer programs.

[0040] A computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described data processing method.

[0041] Accordingly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when read and executed by a processor of a computer device, causes the computer device to perform the aforementioned data processing method.

[0042] Accordingly, this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the data processing method described above.

[0043] This embodiment of the application accurately locates physical shuffle nodes from the physical operator tree corresponding to the database task, and obtains the number of first partitions of the physical shuffle node and the number of second partitions of the child nodes corresponding to the physical shuffle node. When both the number of first partitions and the number of second partitions are equal to the target value, the shuffle operation corresponding to the physical shuffle node is considered redundant. Therefore, the physical operator tree can be optimized to delete the shuffle operation corresponding to the physical shuffle node in the physical operator tree. Finally, the database task is responded to according to the optimized physical operator tree. The above method can reduce the amount of shuffle data, shorten the execution path and execution time of the database task, reduce the computational load, network bandwidth and disk throughput requirements, and improve the computing performance of the distributed computing engine. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the architecture of a data processing system provided in an embodiment of this application;

[0046] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0047] Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0048] Figure 4A This is a schematic diagram of a Spark SQL execution process provided in an embodiment of this application;

[0049] Figure 4B This is a schematic diagram illustrating the optimization process performed on a physical operator tree according to an embodiment of this application;

[0050] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0051] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0053] To better understand the solutions of the embodiments of this application, the relevant terms and concepts that may be involved in the embodiments of this application will be introduced below.

[0054] I. Partition: A partition is the basic unit of data used in Spark job execution. Each partition represents a collection of data, and it is the granularity at which Spark performs distributed computing within a cluster. Partitioning allows Spark to distribute computing tasks across different compute nodes or CPU cores for parallel processing, thereby improving computing performance.

[0055] II. Data Shuffle: Data shuffle refers to the process where data is transferred from one node to another, or from one partition to another, when an operation requires redistributing data (e.g., grouping or merging data by a key). The shuffle process involves sorting, grouping, and repartitioning data, all of which incur significant overhead.

[0056] III. Logical Plan: A logical plan is an abstract representation of a query. It describes the semantics and operations of an SQL query, but does not involve how it is physically executed at the underlying level. The logical plan only focuses on the logical structure of the query, without considering specific execution strategies and optimization methods.

[0057] IV. Physical Plan: The physical plan is a more detailed execution plan that describes how the SQL query will actually be executed, determining how operations such as data scanning, filtering, sorting, and aggregation will be performed. The physical plan covers the specifics of how these operations will be executed and considers details of query execution, such as the algorithms used and the data storage format.

[0058] The data processing system provided in the embodiments of this application will now be described in conjunction with the accompanying drawings. The data processing system is suitable for implementing the data processing method provided in the embodiments of this application.

[0059] Please see Figure 1 This figure is a schematic diagram of the architecture of a data processing system provided in an embodiment of this application. The data processing system may specifically include a terminal device 101 and a server 102. The terminal device 101 and the server 102 are connected via a network, such as a local area network (LAN), a wide area network (WAN), or the mobile internet.

[0060] Terminal equipment 101 may also be referred to as terminal, user equipment (UE), access terminal, user unit, mobile device, user terminal, wireless communication equipment, user agent, or user device. Terminal equipment may be a smart home appliance, a handheld device with wireless communication capabilities (such as a smartphone or tablet), a computing device (such as a personal computer (PC)), an in-vehicle terminal, a smart voice interaction device, a wearable device, or other smart devices, but is not limited to these.

[0061] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0062] In one possible implementation, a user can generate database tasks, such as sorting and aggregation, through a terminal device 101. These tasks can be encapsulated using Structured Query Language (SQL), Application Programming Interface (API), or similar methods. The terminal device 101 can then upload the database tasks to the server 102 via a network protocol, such as Hypertext Transfer Protocol (HTTP).

[0063] In one possible implementation, server 102 can perform logical plan parsing on the database task to obtain a logical operator tree, and further perform physical optimization on the logical operator tree to obtain a physical operator tree. Then, server 102 can determine the physical shuffle node (the physical shuffle node refers to the shuffle node in the physical operator tree, such as the ShuffleExchangeExec node) from the physical operator tree, and obtain the number of the first partition recorded by the physical shuffle node, and the number of the second partition recorded by the child nodes of the physical shuffle node. The number of the first partition indicates the total number of partitions of the output data of the physical shuffle node, and the number of the second partition indicates the total number of partitions of the output data of the child nodes (such as downstream operators) of the physical shuffle node. When the number of the first partition and the number of the second partition are both equal to the target value (such as equal to 1), server 102 can perform optimization processing on the physical operator tree, such as deleting the shuffle operation (such as the shuffle operation) corresponding to the physical shuffle node in the physical operator tree, thereby reducing shuffle data and improving the computing performance of the distributed computing engine. After this, server 102 can respond to the database task according to the optimized physical operator tree, that is, decompose the database task into atomic operations that can be executed in parallel through the physical operator tree, thereby completing the corresponding data processing.

[0064] In one possible implementation, server 102 can return the response results of database tasks to terminal device 101 via a network protocol (such as HTTP protocol), ensuring that terminal device 101 can receive the response results corresponding to each database task.

[0065] It is understood that the system architecture diagrams described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. For example, the data processing method provided in the embodiments of this application can be executed not only by server 102, but also by other servers or server clusters that are different from server 102 and can communicate with terminal device 101 and / or server 102. Those skilled in the art will understand that... Figure 1 The number of terminal devices and servers shown is merely illustrative. Any number of terminal devices and servers can be configured according to business needs. Furthermore, as system architecture evolves and new business scenarios emerge, the technical solutions provided in this application are also applicable to similar technical problems. In subsequent embodiments, "terminal device" will refer to the aforementioned terminal device 101, and "server" will refer to the aforementioned server 102; further details will not be repeated in subsequent embodiments.

[0066] The data processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0067] This application provides a data processing method that can be executed by a computer device, such as a computer device that can be... Figure 1 Server 102 in the data processing system shown. For example... Figure 2 As shown, the data processing method may include, but is not limited to, the following steps S201-S204:

[0068] S201. Determine the physical shuffling node from the physical operator tree corresponding to the database task.

[0069] In this embodiment, a database task can refer to an operation request initiated by a user or application targeting the database, such as data query (e.g., SQL query), aggregation, validation (e.g., data validation during a two-table join). A physical operator tree refers to a physical plan generated by a distributed computing engine (e.g., Spark), composed of physical operators corresponding to specific execution steps. The physical operator tree defines the actual execution logic of the database task in the cluster. A physical shuffling node refers to the physical operator in the physical operator tree responsible for data shuffling (also called data repartitioning) (e.g., Spark's ShuffleExchangeExec node). Data shuffling can redistribute data to different nodes through shuffling operations (e.g., Shuffle operations) to meet the partitioning requirements of subsequent operations (e.g., filtering, aggregation, joining, pruning, etc.).

[0070] In this embodiment, the computer device can traverse the physical operator tree corresponding to the database task to accurately identify all physical shuffle nodes, thereby clarifying potential optimization targets. For example, the computer device can determine all ShuffleExchangeExec nodes in the physical operator tree; these nodes are generally generated by wide dependency operations (such as Group by and Join operations).

[0071] S202. Obtain the number of first partitions recorded by the physical shuffle node, and obtain the number of second partitions recorded by the child nodes of the physical shuffle node.

[0072] In this embodiment, the first partition number indicates the total number of partitions in the output data of the physical shuffling node, and the second partition number indicates the total number of partitions in the output data of the child nodes (such as downstream operators) of the physical shuffling node. Both the first and second partition numbers are dynamic attributes in the physical operator tree, and the computer device can directly obtain the first and second partition numbers from the physical operator tree.

[0073] S203. If the number of the first partition is equal to the target value and the number of the second partition is equal to the target value, then perform optimization processing on the physical operator tree; the optimization processing includes: deleting the shuffling operation corresponding to the physical shuffling node in the physical operator tree.

[0074] In this embodiment, when the number of the first partition and the number of the second partition are equal to the target value, the shuffling operations corresponding to the physical shuffling nodes can be considered redundant. Therefore, the computer device can optimize the shuffling operations corresponding to the physical shuffling nodes in the physical operator tree. Specifically, this optimization can involve deleting shuffling operations, such as removing the physical shuffling nodes from the physical operator tree. This method reduces the amount of shuffle data and eliminates redundant data movement operations, thereby significantly reducing network I / O and disk I / O pressure, improving computational performance, and also reducing computational steps and shortening the execution time of database tasks.

[0075] In one possible implementation, having both the first and second partition counts equal to the target value can mean that both the first and second partition counts are equal to 1. In other words, when the number of partitions in the output data of the physical shuffle node and its corresponding child nodes are both 1, the shuffle operation corresponding to the physical shuffle node in the physical operator tree can be deleted (e.g., the physical shuffle node itself can be deleted). This method determines that data does not need to be transmitted across nodes by having both the first and second partition counts be 1, thus directly deleting the shuffle operation. This reduces the amount of shuffle data without affecting normal business operations, eliminates redundant data movement operations, significantly reduces the overhead of network data transmission and disk persistence between distributed nodes, and lowers computational load, network bandwidth, and disk throughput requirements. Simultaneously, it improves the computational performance of the distributed computing engine, greatly shortens the execution time and path of database tasks, ultimately achieving cost reduction and efficiency improvement.

[0076] S204. Respond to the database task according to the optimized physical operator tree.

[0077] In this embodiment, the computer device can respond to database tasks according to an optimized physical operator tree. Specifically, by executing the optimized physical operator tree (such as a plan after deleting redundant shuffling nodes), the data processing corresponding to the database task is completed, and the response result of the database task can be returned to the user or application. This method ensures the smooth execution of database tasks and provides users with timely response results, thus improving the user experience.

[0078] Based on the above embodiments, the beneficial effects of this application are as follows: This application accurately locates physical shuffle nodes from the physical operator tree corresponding to the database task, and obtains the first partition number of the physical shuffle node and the second partition number of the child nodes corresponding to the physical shuffle node; when both the first partition number and the second partition number are equal to the target value, the shuffle operation corresponding to the physical shuffle node is considered redundant. Therefore, optimization processing can be performed on the physical operator tree to delete the shuffle operation corresponding to the physical shuffle node in the physical operator tree; finally, the database task is responded to according to the optimized physical operator tree. The above method can reduce the amount of shuffle data, shorten the execution path and execution time of the database task, reduce the computational load, network bandwidth and disk throughput requirements, and improve the computational performance of the distributed computing engine.

[0079] This application provides another data processing method, which can be executed by a computer device, such as a computer device that can be... Figure 1 Server 102 in the data processing system shown. For example... Figure 3 As shown, the data processing method may include, but is not limited to, the following steps S301-S306:

[0080] S301. Determine the physical shuffling node from the physical operator tree corresponding to the database task.

[0081] S302. Obtain the number of the first partition recorded by the physical shuffle node, and obtain the number of the second partition recorded by the child nodes of the physical shuffle node.

[0082] The specific implementation methods of steps S301-S302 are described in the relevant descriptions of steps S201-S202 in the foregoing embodiments, and will not be repeated here.

[0083] The following steps, S303-S305, will introduce the specific implementation method for performing optimization processing on the physical operator tree.

[0084] S303. If the number of the first partition is equal to the target value and the number of the second partition is equal to the target value, then determine the parent node of the data shuffling node from the physical operator tree.

[0085] S304. Delete the data shuffling node in the physical operator tree, and delete the node connection relationships between the data shuffling node and its parent and child nodes.

[0086] S305. Establish node connection relationships between parent and child nodes.

[0087] In steps S303-S305 above, when the number of the first partition and the number of the second partition are both equal to the target value, the computer device can first determine the parent node of the data shuffle node from the physical operator tree, thereby clarifying the upstream dependency of the shuffle node and avoiding data link breakage caused by directly deleting the data shuffle node later, thus ensuring the integrity of the data link. Then, the computer device deletes the data shuffle node in the physical operator tree and the node connection relationship between the data shuffle node and its parent node and child node, thereby completely removing redundant Shuffle operations, shortening the execution time and execution path of the database task, and completing resource release. Finally, a new node connection relationship between the parent node and the child node is re-established, ensuring the integrity of the data link.

[0088] In one possible implementation, the computer device may also perform the following steps before performing optimization processing on the physical operator tree:

[0089] (1) When the number of the first partition is equal to the target value and the number of the second partition is equal to the target value, obtain the shuffling field of the physical shuffling node and the partition field indicated by the database partitioning rule corresponding to the database task.

[0090] In this embodiment, the shuffle field refers to the field used by the physical shuffle node (such as ShuffleExchangeExec) when performing data repartitioning. For example, the shuffle field can be "school," indicating that data is repartitioned according to schools. The database partitioning rule corresponding to the database task refers to the physical partitioning strategy adopted by the database table in storing the data involved in the database task. The partitioning field indicated by the database partitioning rule refers to the field used for partitioning in the physical partitioning strategy. For example, the partitioning field can be "school," indicating that data is stored in partitions according to schools.

[0091] (2) If the shuffle field matches the partition field, the step of performing optimization processing on the physical operator tree is triggered.

[0092] In steps (1)-(2) above, before deleting a physical shuffle node, the computer device needs to verify whether the shuffle field matches the partition field. If they match, it means that the data has been distributed on the nodes according to the shuffle field (i.e., the shuffle field matches the partition field), and the number of the first partition and the number of the second partition are both equal to the target value. In this case, the Shuffle operation corresponding to the physical shuffle node is redundant, and the physical shuffle node can be deleted. If they do not match, it means that the data has not been distributed on the nodes according to the shuffle field (i.e., the shuffle field does not match the partition field). In this case, the Shuffle operation corresponding to the physical shuffle node needs to be retained, and the physical shuffle node cannot be deleted. The above method can reduce the amount of Shuffle data while ensuring that the normal operation of the business is not affected.

[0093] For example, when the shuffle field of the physical shuffle node indicates shuffling by class, and the partition field of the database partitioning rule corresponding to the database task indicates partitioning by school, the shuffle field and partition field do not match, so the shuffle operation cannot be omitted; when the shuffle field of the physical shuffle node indicates shuffling by school, and the partition field of the database partitioning rule corresponding to the database task indicates partitioning by school, the shuffle field and partition field match, so the shuffle operation can be omitted.

[0094] In one possible implementation, the more shuffling fields there are and the finer the grouping dimensions, the easier it is to omit shuffling operations. For example, suppose a city has 1000 schools with a total of 50,000 classes, and the physical operator tree includes shuffling operations for both schools and classes. Then, the computer can determine which shuffling operations need to be omitted from the shuffling operations corresponding to multiple shuffling fields based on the partitioning field.

[0095] The first scenario: If the partition field indicates that the data has been partitioned by school, then the shuffling operation for schools in the physical operator tree can be omitted. The second scenario: If the partition field indicates that the data has been partitioned by class, then the shuffling operation for classes in the physical operator tree can be omitted. The third scenario: If the partition field indicates that the data has been partitioned by both class and school, then the shuffling operations for both class and school in the physical operator tree can be omitted (depending on the actual business situation, you can flexibly choose to omit one or both of the shuffling operations for classes and schools). Joint partitioning (also called multi-level partitioning) refers to dividing data into logical partitions based on multiple field combination rules. For example, first, divide the data into first-level partitions by a primary field (such as School), and then further subdivide each first-level partition by secondary fields (such as Class), forming a tree structure. Joint partitioning allows for finer-grained data distribution, reduces the amount of data in a single partition, and improves query efficiency.

[0096] In one possible implementation, for step (2) above, i.e., if the shuffling field matches the partitioning field, then the step of performing optimization processing on the physical operator tree is triggered, one implementation can be as follows:

[0097] (i) If the shuffle field matches the partition field, then obtain the first data volume of the data source corresponding to the child node of the physical shuffle node.

[0098] In this embodiment of the application, the data source corresponding to the child node of the physical shuffling node refers to the data that the child node of the physical shuffling node needs to process, and the data volume of the data source can be recorded as the first data volume.

[0099] (ii) When the first data volume is less than or equal to the first data volume threshold, the step of performing optimization processing on the physical operator tree is triggered.

[0100] In this embodiment, the first data volume threshold can be set to a large value, such as 100G. The optimization process for the physical operator tree is only triggered when the first data volume is less than or equal to the first data volume threshold (which can be considered a data volume upper limit). The above method considers that processing large amounts of data in a single partition can easily lead to performance issues. Therefore, the Shuffle operation is omitted only when the data volume is less than the upper limit, thereby controlling risk through a threshold and ensuring a low performance failure rate while reducing the amount of Shuffled data. It should be noted that the first data volume threshold can be flexibly set according to the actual business operation situation. For example, the first data volume threshold can be a dynamically adjusted value determined based on the actual business operation situation.

[0101] When the first data volume exceeds the first data volume threshold, there is no need to perform optimization processing on the physical operator tree, thereby preserving the original database task processing flow as much as possible, ensuring that the structure of the physical plan is completely consistent with the original logic, avoiding subsequent operator logic errors or performance degradation, avoiding the uncertainty and potential errors introduced by forcibly optimizing the shuffle operation, and thus ensuring the stability of the database task processing.

[0102] In one possible implementation, optimizing the physical operator tree can refer to deleting the shuffling operation corresponding to the physical shuffling node in the physical operator tree and adding an optimized node, which is equivalent to replacing the physical shuffling node with an optimized node. Here, an optimized node refers to a specific operator used to optimize the processing between the parent and child nodes of the original physical shuffling node (i.e., performance improvement is achieved solely by optimizing intermediate processing logic). By replacing the physical shuffling node with an optimized node, effects such as improving node execution efficiency, eliminating computational redundancy, and dynamically balancing task load can typically be achieved. For example, optimized nodes can be used to perform: merging duplicate operations, predicate pushdown (a technique to reduce the amount of query data), pruning, local sorting, etc.

[0103] Therefore, based on this, one implementation of the optimization process for the physical operator tree in the above embodiments can be as follows: determine the parent node of the data shuffling node from the physical operator tree; delete the data shuffling node from the physical operator tree, and the node connection relationships between the data shuffling node and its parent and child nodes respectively; determine the optimization node; establish the node connection relationships between the optimization node and its parent node, and between the optimization node and its child nodes. The optimization node can be determined based on the parent and child nodes of the data shuffling node. The specific implementation of the above process can refer to the method for modifying node connection relationships when deleting the physical shuffling node in the aforementioned embodiments, and will not be repeated here.

[0104] In one possible implementation, before performing optimization on the physical operator tree, if a physical shuffle node in the physical operator tree is connected to an associated shuffle node (the associated shuffle node is a downstream operator of the physical shuffle node), then the physical shuffle node and the associated shuffle node can be merged to obtain a merged shuffle data node. Then, the node connection relationship between the merged shuffle data node and the parent node of the physical shuffle node, as well as the node connection relationship between the merged shuffle data node and the child nodes of the associated shuffle node, can be updated.

[0105] For example, assuming the physical operator subtree includes at least node A, a physical shuffle node, an associated shuffle node, and node B connected sequentially, where both the physical shuffle node and the associated shuffle node are Shuffle nodes, then the physical shuffle node and the associated shuffle node can be merged into a single node. It should be noted that the number of associated shuffle nodes for a physical shuffle node can be one or more; therefore, the physical shuffle node and the associated shuffle node can be uniformly merged into a single merged shuffle data node. This method, by pre-merging shuffle nodes with similar functions, can reduce the number of Shuffle operations in data processing, simplify the complexity of the physical plan, accelerate data processing, and improve overall execution efficiency.

[0106] S306. Respond to database tasks according to the optimized physical operator tree.

[0107] The specific implementation of step S306 is described in the relevant description of step S204 in the foregoing embodiments, and will not be repeated here.

[0108] The method for generating the physical operator tree corresponding to the database task will be described below. In one possible implementation, the computer device can generate the physical operator tree corresponding to the database task as follows:

[0109] (1) Determine the logical shuffling node in the logical operator tree corresponding to the database task.

[0110] In this embodiment, the logical operator tree refers to the logical plan generated by a distributed computing engine (such as Spark), which consists of a series of unoptimized logical operators. The logical operator tree describes the computational logic of the database task but is not bound to a specific execution strategy. The logical operator tree is generated in the logical plan phase, and the physical operator tree is generated in the physical plan phase (the next phase after the logical plan phase). The logical shuffle node refers to the logical operator in the logical operator tree that is responsible for marking the logical operators that need to perform data shuffle (also known as data repartitioning) (such as Spark's ShuffleExchange node).

[0111] In this embodiment, the computer device can traverse the logical operator tree corresponding to the database task to accurately identify all logical shuffle nodes, thereby clarifying potential optimization targets. For example, the computer device can determine all ShuffleExchange nodes in the logical operator tree.

[0112] (2) Obtain the number of third partitions recorded by the logical shuffle node, and obtain the number of fourth partitions recorded by the child nodes of the logical shuffle node.

[0113] In this embodiment, the number of third partitions indicates the total number of partitions of the expected output data of the logical shuffle node, and the number of second partitions indicates the total number of partitions of the expected output data of the child nodes (such as downstream operators) of the logical shuffle node. Both the number of third and fourth partitions are dynamic attributes in the logical operator tree, and the computer device can directly obtain the number of third and fourth partitions from the logical operator tree.

[0114] (3) When the number of at least one of the third partition and the fourth partition is not equal to the target value, obtain the second data volume of the data source corresponding to the child node of the logical shuffle node.

[0115] In this embodiment, the data source corresponding to the child node of the logical shuffling node refers to the data that the child node of the logical shuffling node needs to process, and the data volume of this data source can be denoted as the second data volume. The target value can be set to 1.

[0116] (4) If the second data volume is less than or equal to the second data volume threshold, then adjust the number of partitions in the third and fourth partitions that are not equal to the target value to the target value.

[0117] In this embodiment, the second data volume threshold can be set to a small value, such as 1MB. When the second data volume is less than or equal to the second data volume threshold (which can be considered a lower limit for data volume), the number of partitions in the third and fourth partition counts that are not equal to the target value is adjusted to the target value. For example, when the number of third partitions recorded by the logical shuffle node is 3, it can be changed to 1 under the above conditions. The above method dynamically merges multi-partition shuffle operations in small data volume scenarios into single-partition shuffle operations, which can ensure execution efficiency and maximize resource utilization efficiency (such as reducing redundant memory occupation, optimizing CPU utilization, and saving storage resources). It should be noted that the second data volume threshold can be flexibly set according to the actual business operation situation. For example, the second data volume threshold can be a dynamically adjusted value determined based on the actual business operation situation.

[0118] (5) Generate the physical operator tree corresponding to the database task based on the adjusted logical operator tree.

[0119] The specific implementation of generating the physical operator tree from the adjusted logical operator tree is not the main inventive point of this application, and therefore will not be described further.

[0120] In this embodiment of the application, the number of the third and fourth partitions in the adjusted logical operator tree obtained by the above method are both target values ​​(e.g., equal to 1). Then, the number of the first and second partitions in the physical operator tree generated based on the adjusted logical operator tree are also both 1. At this time, the precondition for performing optimization processing on the physical operator tree in this scheme is met, thereby increasing the possibility of performing optimization processing on the physical operator tree and increasing the applicability of this scheme.

[0121] In one possible implementation, the first data volume threshold and the second data volume threshold in this embodiment can be dynamically adjusted according to actual business conditions. Based on this, the computer device can also perform the following steps:

[0122] (1) Obtain system operation data of the distributed computing engine in real time.

[0123] In one possible implementation, system operation data includes one or more of available memory, CPU utilization, and network load. For example, computer devices can periodically (e.g., every 5 seconds) collect these metrics through a cluster monitoring system (such as Prometheus, Ganglia, etc.) or the built-in API of a distributed computing engine (such as Spark's MetricsSystem). Available memory can include the total amount of currently available memory in the cluster, CPU utilization can include the average CPU utilization of the cluster, and network load can include the current network bandwidth utilization (e.g., the percentage of maximum bandwidth used for transmission).

[0124] (2) Dynamically adjust the target data volume threshold based on system operation data.

[0125] In this embodiment, the target data volume threshold may include one or more of a first data volume threshold and a second data volume threshold. That is, the computer device can set at least one of the first data volume threshold and the second data volume threshold as a dynamically adjustable value, rather than a fixed value. By dynamically adjusting the target data volume threshold using real-time system operation data, the judgment logic regarding omitting the Shuffle operation is aligned with the actual resource status of the distributed computing engine, ensuring the rationality of the first data volume threshold and the second data volume threshold, and avoiding resource waste.

[0126] In one possible implementation, the computer device can perform index normalization processing on different types of system operation data. Then, it can define the weights corresponding to each index according to business needs, and calculate the target data volume threshold (such as the first data volume threshold and the second data volume threshold) based on each index and its corresponding weight.

[0127] In one possible implementation, the computer device can set a fluctuation threshold for dynamically adjusting the target data volume threshold, such as 5%. For example, the computer device can periodically calculate a new target data volume threshold based on system operation data from the distributed computing engine, and then compare the new target data volume threshold with the currently used target data volume threshold. If the fluctuation of the new target data volume threshold compared to the original target data volume threshold is greater than or equal to 5%, the new target data volume threshold is adopted as the required target data volume threshold; if the fluctuation of the new target data volume threshold compared to the original target data volume threshold is less than 5%, the original target data volume threshold continues to be used. By setting a fluctuation threshold, the target data volume threshold is updated only when the actual resource status changes significantly, avoiding task execution failures caused by frequent adjustments to the target data volume threshold.

[0128] In one possible implementation, the data processing method provided in this application embodiment can be encapsulated as a pluggable shuffle optimization rule (the shuffle optimization rule can be referred to as a coalesce shuffle rule). If the user needs to use the shuffle optimization rule, the user can directly configure the shuffle optimization rule in Spark's configuration manager (Spark Conf).

[0129] Based on this, the computer device can also perform the following steps: In the processing stage of the physical operator optimization layer (also called the physical plan optimization layer, which is mainly responsible for converting logical plans into physical plans) of the distributed computing engine, if the physical operator optimization layer is configured with shuffling optimization rules, then the step of determining the physical shuffling node from the physical operator tree corresponding to the database task is triggered, that is, executing steps S201-S204 in the embodiments of this application, or executing steps S301-S306. Users can enable or disable shuffling optimization rules through corresponding configuration switches, thereby achieving strategic and rapid switching for different business scenarios (manifested as whether to use shuffling optimization rules in different scenarios).

[0130] Based on this, the optimization processing of the physical operator tree in the aforementioned embodiments specifically refers to performing optimization processing on the physical operator tree according to the shuffling optimization rules configured in the physical operator optimization layer. Thus, the optimization processing of the physical operator tree can be automatically completed through the pre-configured shuffling optimization rules. The specific implementation method can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0131] The data processing method proposed in this application can be seen as a high-performance optimization scheme for distributed computing engines. It determines whether Shuffle operations need optimization based on physical partitions, primarily applicable to scenarios with numerous Shuffle operations. This effectively reduces the amount of Shuffle data in the distributed computing engine, decreasing CPU computation, cluster data exchange, and read / write operations, as well as reducing computational load, network bandwidth, and disk requirements, while also improving the execution efficiency of database tasks. Furthermore, this scheme possesses strong versatility and adaptability, applicable to various types of distributed architectures or engines (such as MPP architecture), data warehouses, databases, data lakes, and more.

[0132] The following section will introduce Spark SQL query plans. For example... Figure 4A As shown, Figure 4A This is a schematic diagram of a Spark SQL execution process provided in an embodiment of this application, which mainly includes the following flow:

[0133] 1. SQL parsing: Parses SQL statements into an unresolved logical plan.

[0134] The above parsing process involves using an encapsulated logical plan (Dataset), which may include a logical plan tree (LogicalPlan), an execution context (QueryExecution), etc.

[0135] 2. Metadata Binding (Analysis): Applying metadata (such as table structure and column types) from the data dictionary (Catalog) to convert unresolved logical plans into logical plans.

[0136] 3. Logical Optimization: Applying a rule-based optimizer (such as RBO) to transform the logical plan into an optimized logical plan.

[0137] 4. Physical Optimization: Convert the optimized logical plan into a physical plan.

[0138] It should be noted that the shuffling optimization rule in this embodiment is inserted in the physical optimization stage (i.e., the Physical Optimization stage), and the optimization processing of the physical operator tree is implemented in the physical operator optimization layer.

[0139] 5. Cost Model Analysis: The physical plan is analyzed using a cost model to obtain the selected physical plan.

[0140] The cost model is a mechanism in Spark SQL used to evaluate the cost of different physical plans. It can predict the cost of performing an operation based on table statistics (such as the number of rows, the cardinality of columns, etc.), including CPU time, memory usage, and network transmission volume, thereby ensuring that the selected physical plan is theoretically the most economical.

[0141] 6. Code Generation: In this stage, the selected physical plan is transformed into Resilient Distributed Datasets (RDDs). Each RDD includes a series of RDD operations, which are ultimately executed in parallel on the cluster to process the data.

[0142] The data processing method provided in the embodiments of this application will be illustrated below with examples:

[0143] like Figure 4B As shown, Figure 4B This is a schematic diagram illustrating the optimization process performed on a physical operator tree according to an embodiment of this application. It is assumed that the original physical operator tree includes at least node A, a physical shuffle node (such as ShuffleExchangeExec), and node B, wherein node A is connected to the physical shuffle node, and the physical shuffle node is connected to node B.

[0144] When the physical shuffling node is the parent node, node B is a child node of the physical shuffling node. During the optimization process of the physical operator tree, it is necessary to analyze the number of partitions recorded by the physical shuffling node (such as the first number of partitions in this embodiment, which can be represented as ) and the number of partitions recorded by node B (such as the second number of partitions in this embodiment). When both partition counts are 1, the shuffling optimization rule can be triggered. The shuffling optimization rule can delete the physical shuffling node in the physical operator tree. In the optimized physical operator tree, node A is directly connected to node B, while the physical shuffling node is deleted.

[0145] In one possible implementation, a specific way to perform optimization processing on the physical operator tree can be as follows:

[0146] 1. Implement a shuffle optimization rule in the physical operator optimization layer of the distributed computing engine, Physical plan rule: org.apache.spark.sql.execution.CoalesceShuffle.

[0147] 2. When a match is found between (ShffleExchangeExec => Child plan), perform logical judgments on the ShffleExchangeExec node and its child nodes based on the physical partition to determine whether the shuffle optimization rule applies.

[0148] 3. Determine the applicability of the shuffle optimization rules. For the ShuffleExchangeExec node to be applicable to the shuffle optimization rules, both of the following conditions must be met simultaneously:

[0149] (i) Shuffle.partitioning.numPartitions == 1. This condition indicates that the total number of partitions for the output data of the physical shuffle node is 1.

[0150] (ii) Child.outputPartitioning.numPartitions == 1. This condition indicates that the total number of partitions for the output data of the child nodes of the physical shuffle node is 1.

[0151] 4. If the above two conditions are met, the physical operator tree can be optimized by the shuffling optimization rules, thereby completing the transformation of the physical plan and achieving performance optimization.

[0152] The specific implementation methods of the above process are described in the relevant descriptions of the foregoing method embodiments, and will not be repeated here.

[0153] From a code perspective, after optimizing the physical operator tree as provided in this application, the code segment corresponding to the Shuffle operation will be deleted, effectively reducing the number of Shuffle operators, lowering the code size, and optimizing the execution efficiency of SQL queries. This solution not only improves query performance but also optimizes resource utilization, possessing broad application prospects and market value. In the future, this solution can be extended to more distributed computing frameworks, providing a more efficient solution for big data processing.

[0154] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the methods of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0155] Please see Figure 5 This figure is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. This data processing device can be installed in the computer equipment provided in the embodiment of this application, and the computer equipment can be as described above. Figure 1 Server 102 in the data processing system shown. Figure 5The data processing device shown can be a computer program running on a computer device, which can be used to execute... Figure 2 or Figure 3 Some or all of the steps in the method embodiments shown. Please refer to [link / reference]. Figure 5 The data processing apparatus may include the following units:

[0156] The acquisition unit 501 is used to determine the physical shuffling node from the physical operator tree corresponding to the database task;

[0157] The acquisition unit 501 is also used to acquire the number of first partitions recorded by the physical shuffling node and the number of second partitions recorded by the child nodes of the physical shuffling node.

[0158] Execution unit 502 is used to perform optimization processing on the physical operator tree if the number of the first partition is equal to the target value and the number of the second partition is equal to the target value; the optimization processing includes: deleting the shuffling operation corresponding to the physical shuffling node in the physical operator tree;

[0159] Response unit 503 is used to respond to database tasks according to the optimized physical operator tree.

[0160] In one possible implementation, the execution unit 502, when performing optimization processing on the physical operator tree, is specifically used for:

[0161] Determine the parent node of the data shuffling node from the physical operator tree;

[0162] Delete the data shuffling node in the physical operator tree, as well as the node connections between the data shuffling node and its parent and child nodes.

[0163] Establish node connections between parent and child nodes.

[0164] In one possible implementation, the execution unit 502 is further configured to:

[0165] When the number of the first partition is equal to the target value and the number of the second partition is equal to the target value, obtain the shuffling field of the physical shuffling node and the partition field indicated by the database partitioning rule corresponding to the database task;

[0166] If the shuffle field matches the partition field, an optimization step is triggered on the physical operator tree.

[0167] In one possible implementation, the execution unit 502, when used to trigger the optimization process on the physical operator tree if the shuffle field matches the partition field, is specifically used for:

[0168] If the shuffle field matches the partition field, then obtain the first data volume of the data source corresponding to the child node of the physical shuffle node;

[0169] When the first data volume is less than or equal to the first data volume threshold, the step of performing optimization processing on the physical operator tree is triggered.

[0170] In one possible implementation, before determining the physical shuffling node from the physical operator tree corresponding to the database task, the execution unit 502 is further configured to:

[0171] Determine the logical shuffling nodes in the logical operator tree corresponding to the database task;

[0172] Get the number of third partitions recorded by the logical shuffle node, and get the number of fourth partitions recorded by the child nodes of the logical shuffle node.

[0173] When the number of at least one of the third and fourth partitions is not equal to the target value, obtain the second data volume of the data source corresponding to the child node of the logical shuffle node;

[0174] If the second data volume is less than or equal to the second data volume threshold, then adjust the number of partitions in the third and fourth partitions that are not equal to the target value to the target value.

[0175] Generate the physical operator tree corresponding to the database task based on the adjusted logical operator tree.

[0176] In one possible implementation, the execution unit 502 is further configured to:

[0177] Real-time acquisition of system operation data from the distributed computing engine;

[0178] The target data volume threshold is dynamically adjusted based on system operation data; whereby system operation data includes one or more of available memory, CPU utilization, and network load, and the target data volume threshold includes one or more of a first data volume threshold and a second data volume threshold.

[0179] In one possible implementation, the execution unit 502 is further configured to:

[0180] During the processing phase of the physical operator optimization layer of the distributed computing engine, if the physical operator optimization layer is configured with shuffling optimization rules, the step of determining the physical shuffling node from the physical operator tree corresponding to the database task is triggered.

[0181] Specifically, when the execution unit 502 performs optimization processing on the physical operator tree, it is used for:

[0182] The physical operator tree is optimized according to the shuffling optimization rules configured in the physical operator optimization layer.

[0183] In one possible implementation, having the number of the first partition equal to the target value and the number of the second partition equal to the target value means that both the number of the first partition and the number of the second partition are equal to 1.

[0184] It should be noted that the functions of each unit of the data processing device in the embodiments of this application can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the method embodiments of this application, which will not be repeated here.

[0185] According to another embodiment of this application, Figure 5 The data processing apparatus shown can be constructed by combining each unit individually or entirely into one or more other units, or one or more of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the data processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0186] According to another embodiment of this application, the following can be achieved by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), a device capable of performing operations such as... Figure 2 or Figure 3 Computer programs for the steps involved in some or all of the methods shown, to construct, for example... Figure 5 The data processing apparatus shown herein, and the data processing method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable storage medium, loaded onto the aforementioned computing device via the computer-readable storage medium, and executed therein.

[0187] Based on the above methods and apparatus embodiments, this application provides a computer device. Please refer to... Figure 6 This figure is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 6 The computer device shown includes at least a processor 601, an input interface 602, an output interface 603, and a computer-readable storage medium 604. The processor 601, input interface 602, output interface 603, and computer-readable storage medium 604 can be connected via a bus or other means.

[0188] Computer-readable storage medium 604 can be stored in the memory of a computer device. Computer-readable storage medium 604 is used to store computer programs, including program instructions. Processor 601 is used to execute the computer program stored in computer-readable storage medium 604. Processor 601 (or CPU (Central Processing Unit)) is the computing and control core of the computer device; it is suitable for implementing computer programs, specifically for loading and executing computer programs to achieve the above-mentioned functions. Figure 2 or Figure 3 The method flow is shown.

[0189] This application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space for storing the operating system of the computer device. Furthermore, the storage space also stores computer programs suitable for loading and execution by a processor. It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0190] The computer equipment can be the above. Figure 1 The data processing system shown includes server 102. In a specific implementation, processor 601 loads and executes a computer program stored in computer-readable storage medium 604 to implement the corresponding steps of the data processing methods in the various method embodiments of this application. Specifically, the computer program in computer-readable storage medium 604 is loaded by processor 601 and executed with the following steps:

[0191] Determine the physical shuffling nodes from the physical operator tree corresponding to the database task;

[0192] Get the number of the first partition recorded by the physical shuffle node, and get the number of the second partition recorded by the child nodes of the physical shuffle node;

[0193] If the number of the first partition is equal to the target value and the number of the second partition is equal to the target value, then the physical operator tree is optimized. The optimization process includes deleting the shuffle operation corresponding to the physical shuffle node in the physical operator tree.

[0194] The database task is responded to according to the optimized physical operator tree.

[0195] In one possible implementation, the execution unit 502, when performing optimization processing on the physical operator tree, is specifically used for:

[0196] Determine the parent node of the data shuffling node from the physical operator tree;

[0197] Delete the data shuffling node in the physical operator tree, as well as the node connections between the data shuffling node and its parent and child nodes.

[0198] Establish node connections between parent and child nodes.

[0199] In one possible implementation, the processor 601 described above is also used for:

[0200] When the number of the first partition is equal to the target value and the number of the second partition is equal to the target value, obtain the shuffling field of the physical shuffling node and the partition field indicated by the database partitioning rule corresponding to the database task;

[0201] If the shuffle field matches the partition field, an optimization step is triggered on the physical operator tree.

[0202] In one possible implementation, when the processor 601 is used to trigger the optimization process on the physical operator tree if the shuffle field matches the partition field, it is specifically used to:

[0203] If the shuffle field matches the partition field, then obtain the first data volume of the data source corresponding to the child node of the physical shuffle node;

[0204] When the first data volume is less than or equal to the first data volume threshold, the step of performing optimization processing on the physical operator tree is triggered.

[0205] In one possible implementation, before determining the physical shuffling node from the physical operator tree corresponding to the database task, the processor 601 is further configured to:

[0206] Determine the logical shuffling nodes in the logical operator tree corresponding to the database task;

[0207] Get the number of third partitions recorded by the logical shuffle node, and get the number of fourth partitions recorded by the child nodes of the logical shuffle node.

[0208] When the number of at least one of the third and fourth partitions is not equal to the target value, obtain the second data volume of the data source corresponding to the child node of the logical shuffle node;

[0209] If the second data volume is less than or equal to the second data volume threshold, then adjust the number of partitions in the third and fourth partitions that are not equal to the target value to the target value.

[0210] Generate the physical operator tree corresponding to the database task based on the adjusted logical operator tree.

[0211] In one possible implementation, the processor 601 described above is also used for:

[0212] Real-time acquisition of system operation data from the distributed computing engine;

[0213] The target data volume threshold is dynamically adjusted based on system operation data; whereby system operation data includes one or more of available memory, CPU utilization, and network load, and the target data volume threshold includes one or more of a first data volume threshold and a second data volume threshold.

[0214] In one possible implementation, the processor 601 described above is also used for:

[0215] During the processing phase of the physical operator optimization layer of the distributed computing engine, if the physical operator optimization layer is configured with shuffling optimization rules, the step of determining the physical shuffling node from the physical operator tree corresponding to the database task is triggered.

[0216] Specifically, when the processor 601 performs optimization processing on the physical operator tree, it is used for:

[0217] The physical operator tree is optimized according to the shuffling optimization rules configured in the physical operator optimization layer.

[0218] In one possible implementation, having the number of the first partition equal to the target value and the number of the second partition equal to the target value means that both the number of the first partition and the number of the second partition are equal to 1.

[0219] In specific implementations, the processor 601, input interface 602, output interface 603, and computer-readable storage medium 604 described in the embodiments of this application can execute the embodiments of this application. Figure 2 or Figure 3 The implementation methods described in the relevant embodiments of the provided method can also be used to execute the embodiments of this application. Figure 5 The implementation methods described in the relevant embodiments of the provided device will not be repeated here.

[0220] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, systems, and computer devices can be implemented in other ways. The embodiments described above are merely illustrative, and the division of units is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0221] This application also provides a computer program product, which includes program instructions stored in a computer-readable storage medium. A processor of a computer device reads the program instructions from the computer-readable storage medium and executes the program instructions, causing the computer device to perform the aforementioned data processing method, which will not be described in detail here.

[0222] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0223] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more program instructions. When the program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in or transmitted through a computer-readable storage medium. The program instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0224] In this application, data such as database tasks and data sources are involved. When the above embodiments of this application are applied to specific products or technologies, the collection, use and processing of related data should comply with the requirements of relevant laws and regulations. Before collecting data, the information processing rules should be communicated and the individual consent of the subject should be obtained. Data should be processed in strict accordance with the requirements of laws and regulations and personal information processing rules, and technical measures should be taken to ensure the security of related data.

[0225] It should be noted that, in the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0226] It should be noted that the terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature specified with "first" or "second" may explicitly or implicitly include at least one of those features.

[0227] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Determine the physical shuffling nodes from the physical operator tree corresponding to the database task; Obtain the number of first partitions recorded by the physical shuffle node, and obtain the number of second partitions recorded by the child nodes of the physical shuffle node; If the number of the first partitions is equal to the target value and the number of the second partitions is equal to the target value, then an optimization process is performed on the physical operator tree; the optimization process includes: deleting the shuffling operation corresponding to the physical shuffling node in the physical operator tree; The database task is responded to according to the optimized physical operator tree.

2. The method as described in claim 1, characterized in that, The optimization process performed on the physical operator tree includes: Determine the parent node of the data shuffling node from the physical operator tree; Delete the data shuffling node in the physical operator tree, and delete the node connection relationship between the data shuffling node and the parent node and the child node respectively; Establish the node connection relationship between the parent node and the child node.

3. The method as described in claim 1, characterized in that, The method further includes: When the number of the first partition is equal to the target value and the number of the second partition is equal to the target value, obtain the shuffling field of the physical shuffling node and the partition field indicated by the database partitioning rule corresponding to the database task; If the shuffling field matches the partitioning field, then the step of performing optimization processing on the physical operator tree is triggered.

4. The method as described in claim 3, characterized in that, The step of triggering the optimization process on the physical operator tree if the shuffling field matches the partitioning field includes: If the shuffling field matches the partitioning field, then obtain the first data volume of the data source corresponding to the child node of the physical shuffling node; When the first data volume is less than or equal to the first data volume threshold, the step of performing optimization processing on the physical operator tree is triggered.

5. The method as described in claim 1, characterized in that, Before determining the physical shuffling node from the physical operator tree corresponding to the database task, the process also includes: Determine the logical shuffling nodes in the logical operator tree corresponding to the database task; Obtain the number of third partitions recorded by the logical shuffle node, and obtain the number of fourth partitions recorded by the child nodes of the logical shuffle node. When the number of at least one of the third partitions and the fourth partitions is not equal to the target value, obtain the second data volume of the data source corresponding to the child node of the logical shuffle node; If the second data volume is less than or equal to the second data volume threshold, then the number of partitions in the third partition volume and the fourth partition volume that are not equal to the target value will be adjusted to the target value. The physical operator tree corresponding to the database task is generated based on the adjusted logical operator tree.

6. The method as described in claim 4 or 5, characterized in that, The method further includes: Real-time acquisition of system operation data from the distributed computing engine; The target data volume threshold is dynamically adjusted based on the system operation data; wherein the system operation data includes one or more of available memory, CPU utilization, and network load, and the target data volume threshold includes one or more of a first data volume threshold and a second data volume threshold.

7. The method according to any one of claims 1-5, characterized in that, The method further includes: In the processing stage of the physical operator optimization layer of the distributed computing engine, if the physical operator optimization layer is configured with shuffling optimization rules, the step of determining the physical shuffling node from the physical operator tree corresponding to the database task is triggered. The optimization process performed on the physical operator tree includes: The physical operator tree is optimized according to the shuffling optimization rules configured in the physical operator optimization layer.

8. The method according to any one of claims 1-5, characterized in that, The statement that the number of the first partition is equal to the target value and the number of the second partition is equal to the target value means that both the number of the first partition and the number of the second partition are equal to 1.

9. A data processing apparatus, characterized in that, The device includes: The acquisition unit is used to determine the physical shuffling node from the physical operator tree corresponding to the database task; The acquisition unit is further configured to acquire the number of first partitions recorded by the physical shuffling node, and to acquire the number of second partitions recorded by the child nodes of the physical shuffling node. An execution unit is configured to perform optimization processing on the physical operator tree if the number of the first partitions is equal to the target value and the number of the second partitions is equal to the target value; the optimization processing includes deleting the shuffling operation corresponding to the physical shuffling node in the physical operator tree; A response unit is used to respond to the database task according to the optimized physical operator tree.

10. A computer device, characterized in that, The computer device includes: A processor is a tool for implementing computer programs. A computer-readable storage medium storing a computer program adapted to be loaded by the processor and executed as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1-8.

Images